2026-02-25T08:03:17.903 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-25T08:03:17.918 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:03:17.996 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062 branch: wip-logstoremanager description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} thrashers/{default} thrashosds-health workloads/snaps-few-objects-balanced} email: null first_in_suite: false flavor: debug job_id: '69062' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-logstoremanager ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_pg_log_trim_min: 0 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: seastore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd target transaction size: 5 seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Degraded data redundancy - is down - osds down - pg .*? is .*?degraded.*?, acting - pg .*? is stuck - pg degraded - PG_BACKFILL_FULL - Low space hindering backfill .*? backfill_toofull - OSD_ROOT_DOWN - pgs degraded - pgs undersized - is active.*backfill_toofull.* sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: debug sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 thrashosds: min_in: 4 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: c32469671c6b0222f786dd36c34eb702f780a11d owner: scheduled_matan@soko04.front.sepia.ceph.com priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 869 sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: c32469671c6b0222f786dd36c34eb702f780a11d targets: trial039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjhMoFPbq1QANy10SGc4gKqFPs0CVgL+6BZgqTeSLWdZByXSM2OO0fVLvTRc8eeP2WqJ+gF4mmVH28IlFP39H0= trial091.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBItTSov8FyRFZssGD3WHztT8dQGGRGbuj77KeaGVWPcgigCuDtJUoAbThvBCEn4VBYo4zfd63lee9wUFCGANM0s= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_trim_stale_osdmaps: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: balance_reads: true clients: - client.0 objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-25_07:55:35 tube: trial user: matan verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T08:03:17.996 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa; will attempt to use it 2026-02-25T08:03:17.996 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks 2026-02-25T08:03:17.996 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T08:03:17.998 INFO:teuthology.task.internal:Checking packages... 2026-02-25T08:03:17.998 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '6aeb6975bc9a4790a399c9cc70dba9635d2a7819' 2026-02-25T08:03:17.998 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T08:03:17.998 INFO:teuthology.packaging:ref: None 2026-02-25T08:03:17.998 INFO:teuthology.packaging:tag: None 2026-02-25T08:03:17.998 INFO:teuthology.packaging:branch: wip-logstoremanager 2026-02-25T08:03:17.998 INFO:teuthology.packaging:sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:03:17.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-logstoremanager 2026-02-25T08:03:18.121 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5338.g6aeb6975 2026-02-25T08:03:18.123 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T08:03:18.143 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T08:03:18.143 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T08:03:18.157 INFO:teuthology.task.internal:Saving configuration 2026-02-25T08:03:18.172 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T08:03:18.185 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T08:03:18.262 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial039.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 07:57:41.515667', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjhMoFPbq1QANy10SGc4gKqFPs0CVgL+6BZgqTeSLWdZByXSM2OO0fVLvTRc8eeP2WqJ+gF4mmVH28IlFP39H0='} 2026-02-25T08:03:18.320 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial091.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 07:57:41.515057', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBItTSov8FyRFZssGD3WHztT8dQGGRGbuj77KeaGVWPcgigCuDtJUoAbThvBCEn4VBYo4zfd63lee9wUFCGANM0s='} 2026-02-25T08:03:18.320 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T08:03:18.333 INFO:teuthology.task.internal:roles: ubuntu@trial039.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2026-02-25T08:03:18.333 INFO:teuthology.task.internal:roles: ubuntu@trial091.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2026-02-25T08:03:18.333 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T08:03:18.473 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x75198fd10900>, signals=[15]) 2026-02-25T08:03:18.473 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T08:03:18.485 INFO:teuthology.task.internal:Opening connections... 2026-02-25T08:03:18.485 DEBUG:teuthology.task.internal:connecting to ubuntu@trial039.front.sepia.ceph.com 2026-02-25T08:03:18.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:03:18.548 DEBUG:teuthology.task.internal:connecting to ubuntu@trial091.front.sepia.ceph.com 2026-02-25T08:03:18.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:03:18.608 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T08:03:18.620 DEBUG:teuthology.orchestra.run.trial039:> uname -m 2026-02-25T08:03:18.635 INFO:teuthology.orchestra.run.trial039.stdout:x86_64 2026-02-25T08:03:18.635 DEBUG:teuthology.orchestra.run.trial039:> cat /etc/os-release 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:NAME="CentOS Stream" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:VERSION="9" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:ID="centos" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:ID_LIKE="rhel fedora" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:VERSION_ID="9" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:PLATFORM_ID="platform:el9" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:ANSI_COLOR="0;31" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:LOGO="fedora-logo-icon" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:HOME_URL="https://centos.org/" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T08:03:18.691 INFO:teuthology.orchestra.run.trial039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T08:03:18.692 INFO:teuthology.lock.ops:Updating trial039.front.sepia.ceph.com on lock server 2026-02-25T08:03:18.762 DEBUG:teuthology.orchestra.run.trial091:> uname -m 2026-02-25T08:03:18.778 INFO:teuthology.orchestra.run.trial091.stdout:x86_64 2026-02-25T08:03:18.779 DEBUG:teuthology.orchestra.run.trial091:> cat /etc/os-release 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:NAME="CentOS Stream" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:VERSION="9" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:ID="centos" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:ID_LIKE="rhel fedora" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:VERSION_ID="9" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:PLATFORM_ID="platform:el9" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:ANSI_COLOR="0;31" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:LOGO="fedora-logo-icon" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:HOME_URL="https://centos.org/" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T08:03:18.834 INFO:teuthology.orchestra.run.trial091.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T08:03:18.835 INFO:teuthology.lock.ops:Updating trial091.front.sepia.ceph.com on lock server 2026-02-25T08:03:18.915 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T08:03:18.928 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T08:03:18.940 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T08:03:18.940 DEBUG:teuthology.orchestra.run.trial039:> test '!' -e /home/ubuntu/cephtest 2026-02-25T08:03:18.942 DEBUG:teuthology.orchestra.run.trial091:> test '!' -e /home/ubuntu/cephtest 2026-02-25T08:03:18.957 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T08:03:18.968 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T08:03:18.969 DEBUG:teuthology.orchestra.run.trial039:> test -z $(ls -A /var/lib/ceph) 2026-02-25T08:03:18.998 DEBUG:teuthology.orchestra.run.trial091:> test -z $(ls -A /var/lib/ceph) 2026-02-25T08:03:19.014 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T08:03:19.173 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T08:03:19.193 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T08:03:19.193 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T08:03:19.193 DEBUG:teuthology.orchestra.run.trial039:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T08:03:19.193 DEBUG:teuthology.orchestra.run.trial091:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T08:03:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:03:19.209 DEBUG:teuthology.orchestra.run.trial091:> uname -r 2026-02-25T08:03:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:03:19.210 DEBUG:teuthology.orchestra.run.trial039:> uname -r 2026-02-25T08:03:19.264 INFO:teuthology.orchestra.run.trial091.stdout:5.14.0-681.el9.x86_64 2026-02-25T08:03:19.264 INFO:teuthology.task.kernel:Running kernel on trial091: 5.14.0-681.el9.x86_64 2026-02-25T08:03:19.264 DEBUG:teuthology.orchestra.run.trial091:> sudo yum install -y kernel 2026-02-25T08:03:19.265 INFO:teuthology.orchestra.run.trial039.stdout:5.14.0-681.el9.x86_64 2026-02-25T08:03:19.265 INFO:teuthology.task.kernel:Running kernel on trial039: 5.14.0-681.el9.x86_64 2026-02-25T08:03:19.265 DEBUG:teuthology.orchestra.run.trial039:> sudo yum install -y kernel 2026-02-25T08:03:20.522 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.9 MB 00:00 2026-02-25T08:03:20.989 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.9 MB 00:01 2026-02-25T08:03:21.396 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - AppStream 78 MB/s | 27 MB 00:00 2026-02-25T08:03:25.298 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 27 MB 00:03 2026-02-25T08:03:25.374 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 8.0 MB 00:01 2026-02-25T08:03:26.520 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 20 kB 00:00 2026-02-25T08:03:27.053 INFO:teuthology.orchestra.run.trial039.stdout:Extra Packages for Enterprise Linux 43 MB/s | 20 MB 00:00 2026-02-25T08:03:28.621 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 8.0 MB 00:00 2026-02-25T08:03:29.816 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 20 kB 00:00 2026-02-25T08:03:30.257 INFO:teuthology.orchestra.run.trial039.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-25T08:03:30.822 INFO:teuthology.orchestra.run.trial091.stdout:Extra Packages for Enterprise Linux 22 MB/s | 20 MB 00:00 2026-02-25T08:03:31.268 INFO:teuthology.orchestra.run.trial039.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T08:03:31.269 INFO:teuthology.orchestra.run.trial039.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T08:03:31.289 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:03:31.290 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:03:31.290 INFO:teuthology.orchestra.run.trial039.stdout: Package Architecture Version Repository Size 2026-02-25T08:03:31.290 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:03:31.290 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:03:31.290 INFO:teuthology.orchestra.run.trial039.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T08:03:31.290 INFO:teuthology.orchestra.run.trial039.stdout:Installing dependencies: 2026-02-25T08:03:31.290 INFO:teuthology.orchestra.run.trial039.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T08:03:31.290 INFO:teuthology.orchestra.run.trial039.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T08:03:31.290 INFO:teuthology.orchestra.run.trial039.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T08:03:31.291 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:03:31.291 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:03:31.291 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:03:31.291 INFO:teuthology.orchestra.run.trial039.stdout:Install 4 Packages 2026-02-25T08:03:31.291 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:03:31.291 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 89 M 2026-02-25T08:03:31.291 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 135 M 2026-02-25T08:03:31.291 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:03:32.578 INFO:teuthology.orchestra.run.trial039.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 919 kB/s | 901 kB 00:00 2026-02-25T08:03:33.182 INFO:teuthology.orchestra.run.trial039.stdout:(2/4): kernel-core-5.14.0-686.el9.x86_64.rpm 11 MB/s | 17 MB 00:01 2026-02-25T08:03:33.649 INFO:teuthology.orchestra.run.trial039.stdout:(3/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 19 MB/s | 40 MB 00:02 2026-02-25T08:03:33.752 INFO:teuthology.orchestra.run.trial039.stdout:(4/4): kernel-modules-core-5.14.0-686.el9.x86_6 26 MB/s | 31 MB 00:01 2026-02-25T08:03:33.754 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:03:33.754 INFO:teuthology.orchestra.run.trial039.stdout:Total 36 MB/s | 89 MB 00:02 2026-02-25T08:03:33.950 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:03:34.023 INFO:teuthology.orchestra.run.trial091.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-25T08:03:34.154 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:03:34.154 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:03:34.539 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:03:34.540 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:03:35.029 INFO:teuthology.orchestra.run.trial091.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T08:03:35.029 INFO:teuthology.orchestra.run.trial091.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T08:03:35.049 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:03:35.051 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:03:35.051 INFO:teuthology.orchestra.run.trial091.stdout: Package Architecture Version Repository Size 2026-02-25T08:03:35.051 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:03:35.051 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:03:35.051 INFO:teuthology.orchestra.run.trial091.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T08:03:35.051 INFO:teuthology.orchestra.run.trial091.stdout:Installing dependencies: 2026-02-25T08:03:35.051 INFO:teuthology.orchestra.run.trial091.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T08:03:35.051 INFO:teuthology.orchestra.run.trial091.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T08:03:35.051 INFO:teuthology.orchestra.run.trial091.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T08:03:35.051 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:03:35.052 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:03:35.052 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:03:35.052 INFO:teuthology.orchestra.run.trial091.stdout:Install 4 Packages 2026-02-25T08:03:35.052 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:03:35.052 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 89 M 2026-02-25T08:03:35.052 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 135 M 2026-02-25T08:03:35.052 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:03:35.669 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:03:35.973 INFO:teuthology.orchestra.run.trial039.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:03:35.980 INFO:teuthology.orchestra.run.trial091.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 1.0 MB/s | 901 kB 00:00 2026-02-25T08:03:35.996 INFO:teuthology.orchestra.run.trial039.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:03:37.991 INFO:teuthology.orchestra.run.trial091.stdout:(2/4): kernel-modules-core-5.14.0-686.el9.x86_6 15 MB/s | 31 MB 00:02 2026-02-25T08:03:38.438 INFO:teuthology.orchestra.run.trial091.stdout:(3/4): kernel-core-5.14.0-686.el9.x86_64.rpm 5.2 MB/s | 17 MB 00:03 2026-02-25T08:03:38.570 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:03:38.604 INFO:teuthology.orchestra.run.trial039.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:03:41.643 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:03:41.650 INFO:teuthology.orchestra.run.trial039.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:41.823 INFO:teuthology.orchestra.run.trial091.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 6.0 MB/s | 40 MB 00:06 2026-02-25T08:03:41.825 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:03:41.825 INFO:teuthology.orchestra.run.trial091.stdout:Total 13 MB/s | 89 MB 00:06 2026-02-25T08:03:42.016 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:03:42.215 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:03:42.215 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:03:42.600 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:03:42.600 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:03:43.701 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:03:43.989 INFO:teuthology.orchestra.run.trial091.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:03:44.011 INFO:teuthology.orchestra.run.trial091.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:03:44.775 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:46.554 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:03:46.587 INFO:teuthology.orchestra.run.trial091.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:03:49.623 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:03:49.629 INFO:teuthology.orchestra.run.trial091.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:52.745 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:02.135 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:02.156 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:02.345 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:02.345 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:04:02.345 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:04:02.346 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:04:02.403 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:02.403 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:04:02.403 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:04:02.403 INFO:teuthology.orchestra.run.trial039.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:02.403 INFO:teuthology.orchestra.run.trial039.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:02.404 INFO:teuthology.orchestra.run.trial039.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T08:04:02.404 INFO:teuthology.orchestra.run.trial039.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:02.404 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:04:02.404 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:04:02.466 DEBUG:teuthology.orchestra.run.trial039:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:02.494 INFO:teuthology.orchestra.run.trial039.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:02.495 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:02.495 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T08:04:02.495 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T08:04:02.495 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T08:04:02.495 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T08:04:02.495 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial039.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T08:04:02.495 DEBUG:teuthology.orchestra.run.trial039:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:02.560 INFO:teuthology.orchestra.run.trial039.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:02.560 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:02.560 DEBUG:teuthology.orchestra.run.trial039:> sudo rpm -qi grub2-tools 2026-02-25T08:04:02.628 INFO:teuthology.orchestra.run.trial039.stdout:Name : grub2-tools 2026-02-25T08:04:02.628 INFO:teuthology.orchestra.run.trial039.stdout:Epoch : 1 2026-02-25T08:04:02.628 INFO:teuthology.orchestra.run.trial039.stdout:Version : 2.06 2026-02-25T08:04:02.628 INFO:teuthology.orchestra.run.trial039.stdout:Release : 120.el9 2026-02-25T08:04:02.628 INFO:teuthology.orchestra.run.trial039.stdout:Architecture: x86_64 2026-02-25T08:04:02.628 INFO:teuthology.orchestra.run.trial039.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:Group : Unspecified 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:Size : 9108611 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:License : GPLv3+ 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:Packager : builder@centos.org 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:Vendor : CentOS 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:Summary : Support tools for GRUB. 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:Description : 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:hardware devices. 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:04:02.629 INFO:teuthology.orchestra.run.trial039.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T08:04:02.630 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T08:04:02.630 INFO:teuthology.task.kernel:Updating grub on trial039 to boot 5.14.0-686.el9.x86_64 2026-02-25T08:04:02.630 DEBUG:teuthology.orchestra.run.trial039:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T08:04:02.693 INFO:teuthology.orchestra.run.trial039.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T08:04:02.695 DEBUG:teuthology.orchestra.run.trial039:> sudo command -v grubby && echo yes || echo no 2026-02-25T08:04:02.758 INFO:teuthology.orchestra.run.trial039.stdout:/sbin/grubby 2026-02-25T08:04:02.759 INFO:teuthology.orchestra.run.trial039.stdout:yes 2026-02-25T08:04:02.760 DEBUG:teuthology.orchestra.run.trial039:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T08:04:02.823 INFO:teuthology.orchestra.run.trial039.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T08:04:02.824 DEBUG:teuthology.orchestra.run.trial039:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T08:04:02.896 DEBUG:teuthology.orchestra.run.trial039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T08:04:03.099 INFO:teuthology.orchestra.run.trial039.stderr:Generating grub configuration file ... 2026-02-25T08:04:03.167 INFO:teuthology.orchestra.run.trial039.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T08:04:03.173 INFO:teuthology.orchestra.run.trial039.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T08:04:03.424 INFO:teuthology.orchestra.run.trial039.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T08:04:03.430 INFO:teuthology.orchestra.run.trial039.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T08:04:03.463 INFO:teuthology.orchestra.run.trial039.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T08:04:03.469 INFO:teuthology.orchestra.run.trial039.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T08:04:03.501 INFO:teuthology.orchestra.run.trial039.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T08:04:03.507 INFO:teuthology.orchestra.run.trial039.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T08:04:03.531 INFO:teuthology.orchestra.run.trial039.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T08:04:03.537 INFO:teuthology.orchestra.run.trial039.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T08:04:03.624 INFO:teuthology.orchestra.run.trial039.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T08:04:03.632 INFO:teuthology.orchestra.run.trial039.stderr:done 2026-02-25T08:04:03.634 DEBUG:teuthology.orchestra.run.trial039:> test -d /sys/firmware/efi 2026-02-25T08:04:03.651 DEBUG:teuthology.orchestra.run.trial039:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T08:04:03.717 INFO:teuthology.orchestra.run.trial039.stdout:/boot/efi/EFI/BOOT 2026-02-25T08:04:03.717 INFO:teuthology.orchestra.run.trial039.stdout:/boot/efi/EFI/centos 2026-02-25T08:04:03.719 DEBUG:teuthology.orchestra.run.trial039:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T08:04:03.786 INFO:teuthology.orchestra.run.trial039.stdout:no 2026-02-25T08:04:03.787 DEBUG:teuthology.orchestra.run.trial039:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T08:04:03.851 INFO:teuthology.orchestra.run.trial039.stdout:yes 2026-02-25T08:04:03.851 DEBUG:teuthology.orchestra.run.trial039:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T08:04:03.916 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial039 2026-02-25T08:04:03.916 DEBUG:teuthology.orchestra.run.trial039:> sudo bash -c ' 2026-02-25T08:04:03.916 DEBUG:teuthology.orchestra.run.trial039:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T08:04:03.916 DEBUG:teuthology.orchestra.run.trial039:> ' 2026-02-25T08:04:10.206 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:10.228 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:10.416 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:10.416 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:04:10.416 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:04:10.416 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:04:10.473 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:10.473 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:04:10.473 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:04:10.473 INFO:teuthology.orchestra.run.trial091.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:10.473 INFO:teuthology.orchestra.run.trial091.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:10.473 INFO:teuthology.orchestra.run.trial091.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T08:04:10.473 INFO:teuthology.orchestra.run.trial091.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:10.473 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:04:10.473 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:04:10.533 DEBUG:teuthology.orchestra.run.trial091:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:10.563 INFO:teuthology.orchestra.run.trial091.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:10.563 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:10.563 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T08:04:10.563 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T08:04:10.563 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T08:04:10.563 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2026-02-25T08:04:10.563 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial091.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T08:04:10.563 DEBUG:teuthology.orchestra.run.trial091:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:10.629 INFO:teuthology.orchestra.run.trial091.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:10.629 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:10.630 DEBUG:teuthology.orchestra.run.trial091:> sudo rpm -qi grub2-tools 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Name : grub2-tools 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Epoch : 1 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Version : 2.06 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Release : 120.el9 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Architecture: x86_64 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Group : Unspecified 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Size : 9108611 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:License : GPLv3+ 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Packager : builder@centos.org 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Vendor : CentOS 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T08:04:10.698 INFO:teuthology.orchestra.run.trial091.stdout:Summary : Support tools for GRUB. 2026-02-25T08:04:10.699 INFO:teuthology.orchestra.run.trial091.stdout:Description : 2026-02-25T08:04:10.699 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:04:10.699 INFO:teuthology.orchestra.run.trial091.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T08:04:10.699 INFO:teuthology.orchestra.run.trial091.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T08:04:10.699 INFO:teuthology.orchestra.run.trial091.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T08:04:10.699 INFO:teuthology.orchestra.run.trial091.stdout:hardware devices. 2026-02-25T08:04:10.699 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:04:10.699 INFO:teuthology.orchestra.run.trial091.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T08:04:10.699 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T08:04:10.699 INFO:teuthology.task.kernel:Updating grub on trial091 to boot 5.14.0-686.el9.x86_64 2026-02-25T08:04:10.700 DEBUG:teuthology.orchestra.run.trial091:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T08:04:10.763 INFO:teuthology.orchestra.run.trial091.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T08:04:10.764 DEBUG:teuthology.orchestra.run.trial091:> sudo command -v grubby && echo yes || echo no 2026-02-25T08:04:10.828 INFO:teuthology.orchestra.run.trial091.stdout:/sbin/grubby 2026-02-25T08:04:10.829 INFO:teuthology.orchestra.run.trial091.stdout:yes 2026-02-25T08:04:10.830 DEBUG:teuthology.orchestra.run.trial091:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T08:04:10.895 INFO:teuthology.orchestra.run.trial091.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T08:04:10.896 DEBUG:teuthology.orchestra.run.trial091:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T08:04:10.969 DEBUG:teuthology.orchestra.run.trial091:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T08:04:11.179 INFO:teuthology.orchestra.run.trial091.stderr:Generating grub configuration file ... 2026-02-25T08:04:11.247 INFO:teuthology.orchestra.run.trial091.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T08:04:11.254 INFO:teuthology.orchestra.run.trial091.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T08:04:11.507 INFO:teuthology.orchestra.run.trial091.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T08:04:11.514 INFO:teuthology.orchestra.run.trial091.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T08:04:11.548 INFO:teuthology.orchestra.run.trial091.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T08:04:11.555 INFO:teuthology.orchestra.run.trial091.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T08:04:11.587 INFO:teuthology.orchestra.run.trial091.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T08:04:11.593 INFO:teuthology.orchestra.run.trial091.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T08:04:11.617 INFO:teuthology.orchestra.run.trial091.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T08:04:11.623 INFO:teuthology.orchestra.run.trial091.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T08:04:11.711 INFO:teuthology.orchestra.run.trial091.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T08:04:11.719 INFO:teuthology.orchestra.run.trial091.stderr:done 2026-02-25T08:04:11.722 DEBUG:teuthology.orchestra.run.trial091:> test -d /sys/firmware/efi 2026-02-25T08:04:11.738 DEBUG:teuthology.orchestra.run.trial091:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T08:04:11.804 INFO:teuthology.orchestra.run.trial091.stdout:/boot/efi/EFI/BOOT 2026-02-25T08:04:11.804 INFO:teuthology.orchestra.run.trial091.stdout:/boot/efi/EFI/centos 2026-02-25T08:04:11.805 DEBUG:teuthology.orchestra.run.trial091:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T08:04:11.870 INFO:teuthology.orchestra.run.trial091.stdout:no 2026-02-25T08:04:11.871 DEBUG:teuthology.orchestra.run.trial091:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T08:04:11.935 INFO:teuthology.orchestra.run.trial091.stdout:yes 2026-02-25T08:04:11.936 DEBUG:teuthology.orchestra.run.trial091:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T08:04:12.000 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial091 2026-02-25T08:04:12.000 DEBUG:teuthology.orchestra.run.trial091:> sudo bash -c ' 2026-02-25T08:04:12.000 DEBUG:teuthology.orchestra.run.trial091:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T08:04:12.000 DEBUG:teuthology.orchestra.run.trial091:> ' 2026-02-25T08:04:33.959 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T08:04:33.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial039.front.sepia.ceph.com' 2026-02-25T08:04:33.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:04:42.042 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2026-02-25T08:04:42.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial091.front.sepia.ceph.com' 2026-02-25T08:04:42.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:04:48.256 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.39 2026-02-25T08:04:56.319 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.91 2026-02-25T08:04:57.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial039.front.sepia.ceph.com' 2026-02-25T08:04:57.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:05:05.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial091.front.sepia.ceph.com' 2026-02-25T08:05:05.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:05:33.280 DEBUG:teuthology.orchestra.run.trial039:> true 2026-02-25T08:05:33.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial039.front.sepia.ceph.com' 2026-02-25T08:05:33.429 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-25T08:05:33.430 DEBUG:teuthology.orchestra.run.trial039:> uname -r 2026-02-25T08:05:33.485 INFO:teuthology.orchestra.run.trial039.stdout:5.14.0-686.el9.x86_64 2026-02-25T08:05:33.486 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T08:05:33.486 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T08:05:33.486 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T08:05:34.486 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T08:05:34.486 DEBUG:teuthology.orchestra.run.trial039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T08:05:34.516 INFO:teuthology.orchestra.run.trial039.stdout:ttyS1 2026-02-25T08:05:34.538 DEBUG:teuthology.parallel:result is None 2026-02-25T08:05:41.475 DEBUG:teuthology.orchestra.run.trial091:> true 2026-02-25T08:05:41.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial091.front.sepia.ceph.com' 2026-02-25T08:05:41.629 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-686.el9.x86_64"... 2026-02-25T08:05:41.629 DEBUG:teuthology.orchestra.run.trial091:> uname -r 2026-02-25T08:05:41.684 INFO:teuthology.orchestra.run.trial091.stdout:5.14.0-686.el9.x86_64 2026-02-25T08:05:41.684 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T08:05:41.684 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T08:05:41.684 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T08:05:42.685 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2026-02-25T08:05:42.685 DEBUG:teuthology.orchestra.run.trial091:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T08:05:42.715 INFO:teuthology.orchestra.run.trial091.stdout:ttyS1 2026-02-25T08:05:42.736 DEBUG:teuthology.parallel:result is None 2026-02-25T08:05:42.736 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T08:05:42.747 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T08:05:42.747 DEBUG:teuthology.orchestra.run.trial039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T08:05:42.749 DEBUG:teuthology.orchestra.run.trial091:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T08:05:42.792 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T08:05:42.804 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T08:05:42.818 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T08:05:42.818 DEBUG:teuthology.orchestra.run.trial039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T08:05:42.820 DEBUG:teuthology.orchestra.run.trial091:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T08:05:42.851 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T08:05:42.863 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T08:05:42.863 DEBUG:teuthology.orchestra.run.trial039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T08:05:42.875 DEBUG:teuthology.orchestra.run.trial091:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T08:05:42.896 INFO:teuthology.orchestra.run.trial039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:05:42.905 INFO:teuthology.orchestra.run.trial039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:05:42.912 INFO:teuthology.orchestra.run.trial091.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:05:42.921 INFO:teuthology.orchestra.run.trial091.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:05:42.922 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T08:05:42.934 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T08:05:42.934 DEBUG:teuthology.orchestra.run.trial039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T08:05:42.949 DEBUG:teuthology.orchestra.run.trial091:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T08:05:42.986 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T08:05:42.999 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T08:05:42.999 DEBUG:teuthology.orchestra.run.trial039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T08:05:43.013 DEBUG:teuthology.orchestra.run.trial091:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T08:05:43.042 DEBUG:teuthology.orchestra.run.trial039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:05:43.141 DEBUG:teuthology.orchestra.run.trial039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:05:43.164 DEBUG:teuthology.orchestra.run.trial039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:05:43.284 DEBUG:teuthology.orchestra.run.trial039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:05:43.308 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:05:43.308 DEBUG:teuthology.orchestra.run.trial039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T08:05:43.371 DEBUG:teuthology.orchestra.run.trial091:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:05:43.454 DEBUG:teuthology.orchestra.run.trial091:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:05:43.478 DEBUG:teuthology.orchestra.run.trial091:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:05:43.601 DEBUG:teuthology.orchestra.run.trial091:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:05:43.624 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:05:43.624 DEBUG:teuthology.orchestra.run.trial091:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T08:05:43.688 DEBUG:teuthology.orchestra.run.trial039:> sudo service rsyslog restart 2026-02-25T08:05:43.690 DEBUG:teuthology.orchestra.run.trial091:> sudo service rsyslog restart 2026-02-25T08:05:43.717 INFO:teuthology.orchestra.run.trial039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:05:43.756 INFO:teuthology.orchestra.run.trial091.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:05:44.282 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T08:05:44.293 INFO:teuthology.task.internal:Starting timer... 2026-02-25T08:05:44.293 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T08:05:44.311 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T08:05:44.324 DEBUG:teuthology.orchestra.run.trial039:> sudo service auditd rotate 2026-02-25T08:05:44.353 INFO:teuthology.orchestra.run.trial039.stdout:Rotating logs: 2026-02-25T08:05:44.354 DEBUG:teuthology.orchestra.run.trial091:> sudo service auditd rotate 2026-02-25T08:05:44.384 INFO:teuthology.orchestra.run.trial091.stdout:Rotating logs: 2026-02-25T08:05:44.386 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T08:05:44.386 DEBUG:teuthology.orchestra.run.trial039:> /usr/sbin/getenforce 2026-02-25T08:05:44.411 INFO:teuthology.orchestra.run.trial039.stdout:Permissive 2026-02-25T08:05:44.411 DEBUG:teuthology.orchestra.run.trial091:> /usr/sbin/getenforce 2026-02-25T08:05:44.442 INFO:teuthology.orchestra.run.trial091.stdout:Permissive 2026-02-25T08:05:44.442 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial039.front.sepia.ceph.com': 'permissive', 'ubuntu@trial091.front.sepia.ceph.com': 'permissive'} 2026-02-25T08:05:44.443 DEBUG:teuthology.orchestra.run.trial039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:05:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:05:44.474 DEBUG:teuthology.orchestra.run.trial091:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:05:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:05:44.506 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T08:05:44.506 DEBUG:teuthology.orchestra.run.trial039:> sudo /usr/sbin/setenforce permissive 2026-02-25T08:05:44.537 DEBUG:teuthology.orchestra.run.trial091:> sudo /usr/sbin/setenforce permissive 2026-02-25T08:05:44.569 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T08:05:44.581 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T08:05:44.587 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T08:05:44.588 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial039.front.sepia.ceph.com,trial091.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T08:08:15.997 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial039.front.sepia.ceph.com'), Remote(name='ubuntu@trial091.front.sepia.ceph.com')] 2026-02-25T08:08:15.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial039.front.sepia.ceph.com' 2026-02-25T08:08:15.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:08:16.084 DEBUG:teuthology.orchestra.run.trial039:> true 2026-02-25T08:08:16.160 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial039.front.sepia.ceph.com' 2026-02-25T08:08:16.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial091.front.sepia.ceph.com' 2026-02-25T08:08:16.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial091.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:08:16.226 DEBUG:teuthology.orchestra.run.trial091:> true 2026-02-25T08:08:16.299 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial091.front.sepia.ceph.com' 2026-02-25T08:08:16.299 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T08:08:16.311 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T08:08:16.312 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T08:08:16.312 DEBUG:teuthology.orchestra.run.trial039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:08:16.314 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T08:08:16.314 DEBUG:teuthology.orchestra.run.trial091:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:08:16.340 INFO:teuthology.orchestra.run.trial039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T08:08:16.352 INFO:teuthology.orchestra.run.trial039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T08:08:16.365 INFO:teuthology.orchestra.run.trial091.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T08:08:16.373 INFO:teuthology.orchestra.run.trial039.stderr:sudo: ntpd: command not found 2026-02-25T08:08:16.378 INFO:teuthology.orchestra.run.trial091.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T08:08:16.382 INFO:teuthology.orchestra.run.trial039.stdout:506 Cannot talk to daemon 2026-02-25T08:08:16.394 INFO:teuthology.orchestra.run.trial039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T08:08:16.400 INFO:teuthology.orchestra.run.trial091.stderr:sudo: ntpd: command not found 2026-02-25T08:08:16.406 INFO:teuthology.orchestra.run.trial039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T08:08:16.410 INFO:teuthology.orchestra.run.trial091.stdout:506 Cannot talk to daemon 2026-02-25T08:08:16.421 INFO:teuthology.orchestra.run.trial091.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T08:08:16.433 INFO:teuthology.orchestra.run.trial091.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T08:08:16.454 INFO:teuthology.orchestra.run.trial039.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:08:16.485 INFO:teuthology.orchestra.run.trial091.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:08:16.499 INFO:teuthology.orchestra.run.trial091.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:08:16.499 INFO:teuthology.orchestra.run.trial091.stdout:=============================================================================== 2026-02-25T08:08:16.499 INFO:teuthology.orchestra.run.trial091.stdout:^? ntp1.glypnod.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:16.499 INFO:teuthology.orchestra.run.trial091.stdout:^? 23.186.168.133 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:16.499 INFO:teuthology.orchestra.run.trial091.stdout:^? 172-233-189-68.ip.linode> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:16.500 INFO:teuthology.orchestra.run.trial091.stdout:^? 23.186.168.132 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:16.579 INFO:teuthology.orchestra.run.trial039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:08:16.579 INFO:teuthology.orchestra.run.trial039.stdout:=============================================================================== 2026-02-25T08:08:16.579 INFO:teuthology.orchestra.run.trial039.stdout:^? kc1cloud.147.72.155.23.m> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:16.579 INFO:teuthology.orchestra.run.trial039.stdout:^? nu.binary.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:16.579 INFO:teuthology.orchestra.run.trial039.stdout:^? 108.61.215.221.vultruser> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:16.579 INFO:teuthology.orchestra.run.trial039.stdout:^? ntp3.radio-sunshine.org 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:16.580 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:08:16.591 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:08:16.591 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:08:16.591 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:08:16.591 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:08:16.593 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:08:16.593 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-25T08:08:16.594 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:08:16.594 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:08:16.595 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:08:16.595 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:08:16.751 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:08:16.751 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:08:16.770 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:08:16.770 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:08:17.119 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:08:17.119 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:08:17.119 DEBUG:teuthology.orchestra.run.trial039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:08:17.144 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T08:08:17.144 DEBUG:teuthology.orchestra.run.trial039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:08:17.161 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:08:17.161 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:08:17.161 DEBUG:teuthology.orchestra.run.trial091:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:08:17.187 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T08:08:17.187 DEBUG:teuthology.orchestra.run.trial091:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:08:17.208 DEBUG:teuthology.orchestra.run.trial039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:08:17.252 DEBUG:teuthology.orchestra.run.trial091:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:08:17.280 DEBUG:teuthology.orchestra.run.trial039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:08:17.325 DEBUG:teuthology.orchestra.run.trial091:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:08:17.344 INFO:teuthology.orchestra.run.trial039.stdout:check_obsoletes = 1 2026-02-25T08:08:17.345 DEBUG:teuthology.orchestra.run.trial039:> sudo yum clean all 2026-02-25T08:08:17.388 INFO:teuthology.orchestra.run.trial091.stdout:check_obsoletes = 1 2026-02-25T08:08:17.389 DEBUG:teuthology.orchestra.run.trial091:> sudo yum clean all 2026-02-25T08:08:17.493 INFO:teuthology.orchestra.run.trial039.stdout:41 files removed 2026-02-25T08:08:17.507 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-radosgw 2026-02-25T08:08:17.536 INFO:teuthology.orchestra.run.trial091.stdout:41 files removed 2026-02-25T08:08:17.551 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-radosgw 2026-02-25T08:08:18.355 INFO:teuthology.orchestra.run.trial039.stdout:ceph packages for x86_64 134 kB/s | 94 kB 00:00 2026-02-25T08:08:18.405 INFO:teuthology.orchestra.run.trial091.stdout:ceph packages for x86_64 133 kB/s | 94 kB 00:00 2026-02-25T08:08:19.016 INFO:teuthology.orchestra.run.trial039.stdout:ceph noarch packages 39 kB/s | 24 kB 00:00 2026-02-25T08:08:19.064 INFO:teuthology.orchestra.run.trial091.stdout:ceph noarch packages 39 kB/s | 24 kB 00:00 2026-02-25T08:08:19.602 INFO:teuthology.orchestra.run.trial039.stdout:ceph source packages 3.7 kB/s | 2.1 kB 00:00 2026-02-25T08:08:19.647 INFO:teuthology.orchestra.run.trial091.stdout:ceph source packages 3.7 kB/s | 2.1 kB 00:00 2026-02-25T08:08:21.270 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.9 MB 00:01 2026-02-25T08:08:21.722 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.9 MB 00:02 2026-02-25T08:08:24.271 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-25T08:08:31.614 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 27 MB 00:09 2026-02-25T08:08:35.172 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 8.0 MB 00:01 2026-02-25T08:08:35.914 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - CRB 884 kB/s | 8.0 MB 00:09 2026-02-25T08:08:36.146 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - Extras packages 92 kB/s | 20 kB 00:00 2026-02-25T08:08:36.400 INFO:teuthology.orchestra.run.trial091.stdout:Extra Packages for Enterprise Linux 108 MB/s | 20 MB 00:00 2026-02-25T08:08:37.109 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 20 kB 00:00 2026-02-25T08:08:37.793 INFO:teuthology.orchestra.run.trial039.stdout:Extra Packages for Enterprise Linux 33 MB/s | 20 MB 00:00 2026-02-25T08:08:39.613 INFO:teuthology.orchestra.run.trial091.stdout:lab-extras 1.3 MB/s | 50 kB 00:00 2026-02-25T08:08:40.832 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:08:40.833 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: Package Arch Version Repository Size 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout:Upgrading: 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout:Installing dependencies: 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T08:08:40.834 INFO:teuthology.orchestra.run.trial091.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T08:08:40.835 INFO:teuthology.orchestra.run.trial091.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout:Installing weak dependencies: 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout:Install 35 Packages 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout:Upgrade 2 Packages 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:08:40.836 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 61 M 2026-02-25T08:08:40.837 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:08:41.024 INFO:teuthology.orchestra.run.trial039.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-25T08:08:42.239 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:08:42.241 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:08:42.241 INFO:teuthology.orchestra.run.trial039.stdout: Package Arch Version Repository Size 2026-02-25T08:08:42.241 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:08:42.241 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:08:42.241 INFO:teuthology.orchestra.run.trial039.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T08:08:42.241 INFO:teuthology.orchestra.run.trial039.stdout:Upgrading: 2026-02-25T08:08:42.241 INFO:teuthology.orchestra.run.trial039.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout:Installing dependencies: 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T08:08:42.242 INFO:teuthology.orchestra.run.trial039.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T08:08:42.243 INFO:teuthology.orchestra.run.trial039.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T08:08:42.243 INFO:teuthology.orchestra.run.trial039.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T08:08:42.243 INFO:teuthology.orchestra.run.trial039.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T08:08:42.243 INFO:teuthology.orchestra.run.trial039.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T08:08:42.243 INFO:teuthology.orchestra.run.trial039.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T08:08:42.243 INFO:teuthology.orchestra.run.trial039.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T08:08:42.243 INFO:teuthology.orchestra.run.trial039.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T08:08:42.243 INFO:teuthology.orchestra.run.trial039.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T08:08:42.243 INFO:teuthology.orchestra.run.trial039.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T08:08:42.243 INFO:teuthology.orchestra.run.trial039.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T08:08:42.244 INFO:teuthology.orchestra.run.trial039.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout:Installing weak dependencies: 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout:Install 35 Packages 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout:Upgrade 2 Packages 2026-02-25T08:08:42.245 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:08:42.246 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 61 M 2026-02-25T08:08:42.246 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:08:47.381 INFO:teuthology.orchestra.run.trial039.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 1.1 MB/s | 5.1 MB 00:04 2026-02-25T08:08:47.592 INFO:teuthology.orchestra.run.trial039.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 119 kB/s | 25 kB 00:00 2026-02-25T08:08:47.793 INFO:teuthology.orchestra.run.trial039.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 184 kB/s | 37 kB 00:00 2026-02-25T08:08:47.935 INFO:teuthology.orchestra.run.trial039.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 181 kB/s | 26 kB 00:00 2026-02-25T08:08:49.822 INFO:teuthology.orchestra.run.trial091.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 628 kB/s | 5.1 MB 00:08 2026-02-25T08:08:49.957 INFO:teuthology.orchestra.run.trial091.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 186 kB/s | 25 kB 00:00 2026-02-25T08:08:50.096 INFO:teuthology.orchestra.run.trial091.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 267 kB/s | 37 kB 00:00 2026-02-25T08:08:50.217 INFO:teuthology.orchestra.run.trial039.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 377 kB/s | 860 kB 00:02 2026-02-25T08:08:50.237 INFO:teuthology.orchestra.run.trial091.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 181 kB/s | 26 kB 00:00 2026-02-25T08:08:50.783 INFO:teuthology.orchestra.run.trial039.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 465 kB/s | 263 kB 00:00 2026-02-25T08:08:52.496 INFO:teuthology.orchestra.run.trial091.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 381 kB/s | 860 kB 00:02 2026-02-25T08:08:53.060 INFO:teuthology.orchestra.run.trial091.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 466 kB/s | 263 kB 00:00 2026-02-25T08:09:00.921 INFO:teuthology.orchestra.run.trial039.stdout:(7/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_6 569 kB/s | 5.6 MB 00:10 2026-02-25T08:09:00.992 INFO:teuthology.orchestra.run.trial039.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 642 kB/s | 45 kB 00:00 2026-02-25T08:09:01.344 INFO:teuthology.orchestra.run.trial039.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 526 kB/s | 185 kB 00:00 2026-02-25T08:09:01.865 INFO:teuthology.orchestra.run.trial039.stdout:(10/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 390 kB/s | 204 kB 00:00 2026-02-25T08:09:03.194 INFO:teuthology.orchestra.run.trial039.stdout:(11/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9 554 kB/s | 11 MB 00:20 2026-02-25T08:09:03.507 INFO:teuthology.orchestra.run.trial039.stdout:(12/37): python3-rados-20.3.0-5338.g6aeb6975.el 197 kB/s | 324 kB 00:01 2026-02-25T08:09:03.929 INFO:teuthology.orchestra.run.trial039.stdout:(13/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 237 kB/s | 100 kB 00:00 2026-02-25T08:09:04.005 INFO:teuthology.orchestra.run.trial039.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 439 kB/s | 33 kB 00:00 2026-02-25T08:09:04.106 INFO:teuthology.orchestra.run.trial039.stdout:(15/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 335 kB/s | 306 kB 00:00 2026-02-25T08:09:04.130 INFO:teuthology.orchestra.run.trial039.stdout:(16/37): nvme-cli-2.16-1.el9.x86_64.rpm 9.2 MB/s | 1.2 MB 00:00 2026-02-25T08:09:04.182 INFO:teuthology.orchestra.run.trial039.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 1.2 MB/s | 93 kB 00:00 2026-02-25T08:09:04.191 INFO:teuthology.orchestra.run.trial039.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 1.7 MB/s | 104 kB 00:00 2026-02-25T08:09:04.216 INFO:teuthology.orchestra.run.trial039.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 6.2 MB/s | 160 kB 00:00 2026-02-25T08:09:04.223 INFO:teuthology.orchestra.run.trial039.stdout:(20/37): libnbd-1.20.3-4.el9.x86_64.rpm 4.0 MB/s | 164 kB 00:00 2026-02-25T08:09:04.241 INFO:teuthology.orchestra.run.trial039.stdout:(21/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.8 MB/s | 45 kB 00:00 2026-02-25T08:09:04.268 INFO:teuthology.orchestra.run.trial039.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 11 MB/s | 292 kB 00:00 2026-02-25T08:09:04.278 INFO:teuthology.orchestra.run.trial039.stdout:(23/37): librdkafka-1.6.1-102.el9.x86_64.rpm 12 MB/s | 662 kB 00:00 2026-02-25T08:09:04.295 INFO:teuthology.orchestra.run.trial039.stdout:(24/37): qatlib-25.08.0-2.el9.x86_64.rpm 8.6 MB/s | 240 kB 00:00 2026-02-25T08:09:04.298 INFO:teuthology.orchestra.run.trial039.stdout:(25/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.7 MB/s | 37 kB 00:00 2026-02-25T08:09:04.319 INFO:teuthology.orchestra.run.trial039.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.7 MB/s | 66 kB 00:00 2026-02-25T08:09:04.404 INFO:teuthology.orchestra.run.trial091.stdout:(7/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_6 508 kB/s | 5.6 MB 00:11 2026-02-25T08:09:04.473 INFO:teuthology.orchestra.run.trial091.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 655 kB/s | 45 kB 00:00 2026-02-25T08:09:04.521 INFO:teuthology.orchestra.run.trial039.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.3 MB/s | 308 kB 00:00 2026-02-25T08:09:04.555 INFO:teuthology.orchestra.run.trial039.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 743 kB/s | 25 kB 00:00 2026-02-25T08:09:04.595 INFO:teuthology.orchestra.run.trial039.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2026-02-25T08:09:04.635 INFO:teuthology.orchestra.run.trial039.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 1.7 MB/s | 67 kB 00:00 2026-02-25T08:09:04.690 INFO:teuthology.orchestra.run.trial039.stdout:(31/37): libarrow-9.0.0-15.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2026-02-25T08:09:04.727 INFO:teuthology.orchestra.run.trial039.stdout:(32/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 8.9 MB/s | 838 kB 00:00 2026-02-25T08:09:04.728 INFO:teuthology.orchestra.run.trial039.stdout:(33/37): re2-20211101-20.el9.x86_64.rpm 4.8 MB/s | 191 kB 00:00 2026-02-25T08:09:04.752 INFO:teuthology.orchestra.run.trial091.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 663 kB/s | 185 kB 00:00 2026-02-25T08:09:04.813 INFO:teuthology.orchestra.run.trial039.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2026-02-25T08:09:04.998 INFO:teuthology.orchestra.run.trial091.stdout:(10/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9 487 kB/s | 11 MB 00:23 2026-02-25T08:09:05.028 INFO:teuthology.orchestra.run.trial091.stdout:(11/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 738 kB/s | 204 kB 00:00 2026-02-25T08:09:05.445 INFO:teuthology.orchestra.run.trial091.stdout:(12/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 733 kB/s | 306 kB 00:00 2026-02-25T08:09:05.654 INFO:teuthology.orchestra.run.trial091.stdout:(13/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 480 kB/s | 100 kB 00:00 2026-02-25T08:09:05.665 INFO:teuthology.orchestra.run.trial091.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 2.9 MB/s | 33 kB 00:00 2026-02-25T08:09:05.686 INFO:teuthology.orchestra.run.trial091.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 55 MB/s | 1.2 MB 00:00 2026-02-25T08:09:05.691 INFO:teuthology.orchestra.run.trial091.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 17 MB/s | 93 kB 00:00 2026-02-25T08:09:05.762 INFO:teuthology.orchestra.run.trial091.stdout:(17/37): python3-rados-20.3.0-5338.g6aeb6975.el 424 kB/s | 324 kB 00:00 2026-02-25T08:09:06.055 INFO:teuthology.orchestra.run.trial091.stdout:[MIRROR] boost-program-options-1.75.0-13.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-13.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:09:06.101 INFO:teuthology.orchestra.run.trial091.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:09:06.415 INFO:teuthology.orchestra.run.trial091.stdout:[MIRROR] boost-program-options-1.75.0-13.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-13.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:09:06.446 INFO:teuthology.orchestra.run.trial091.stdout:[MIRROR] libnbd-1.20.3-4.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libnbd-1.20.3-4.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:09:06.456 INFO:teuthology.orchestra.run.trial091.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 136 kB/s | 104 kB 00:00 2026-02-25T08:09:06.468 INFO:teuthology.orchestra.run.trial091.stdout:(19/37): libnbd-1.20.3-4.el9.x86_64.rpm 232 kB/s | 164 kB 00:00 2026-02-25T08:09:06.976 INFO:teuthology.orchestra.run.trial091.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 89 kB/s | 45 kB 00:00 2026-02-25T08:09:07.113 INFO:teuthology.orchestra.run.trial091.stdout:(21/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 244 kB/s | 160 kB 00:00 2026-02-25T08:09:07.202 INFO:teuthology.orchestra.run.trial091.stdout:[MIRROR] lttng-ust-2.12.0-6.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lttng-ust-2.12.0-6.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:09:07.539 INFO:teuthology.orchestra.run.trial091.stdout:[MIRROR] lttng-ust-2.12.0-6.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/lttng-ust-2.12.0-6.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:09:07.564 INFO:teuthology.orchestra.run.trial091.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 648 kB/s | 292 kB 00:00 2026-02-25T08:09:07.899 INFO:teuthology.orchestra.run.trial091.stdout:[MIRROR] qatlib-25.08.0-2.el9.x86_64.rpm: Status code: 503 for https://ftp2.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-25.08.0-2.el9.x86_64.rpm (IP: 140.211.166.134) 2026-02-25T08:09:07.922 INFO:teuthology.orchestra.run.trial091.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 670 kB/s | 240 kB 00:00 2026-02-25T08:09:07.943 INFO:teuthology.orchestra.run.trial091.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.7 MB/s | 37 kB 00:00 2026-02-25T08:09:07.961 INFO:teuthology.orchestra.run.trial091.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 3.5 MB/s | 66 kB 00:00 2026-02-25T08:09:08.782 INFO:teuthology.orchestra.run.trial091.stdout:(26/37): librdkafka-1.6.1-102.el9.x86_64.rpm 367 kB/s | 662 kB 00:01 2026-02-25T08:09:11.164 INFO:teuthology.orchestra.run.trial039.stdout:(35/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 434 kB/s | 2.7 MB 00:06 2026-02-25T08:09:14.784 INFO:teuthology.orchestra.run.trial039.stdout:(36/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 334 kB/s | 3.3 MB 00:10 2026-02-25T08:09:31.765 INFO:teuthology.orchestra.run.trial039.stdout:(37/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 433 kB/s | 21 MB 00:49 2026-02-25T08:09:31.767 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:09:31.767 INFO:teuthology.orchestra.run.trial039.stdout:Total 1.2 MB/s | 61 MB 00:49 2026-02-25T08:09:31.861 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:09:31.879 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:09:31.879 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:09:32.125 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:09:32.125 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:09:32.417 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:09:32.563 INFO:teuthology.orchestra.run.trial039.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T08:09:32.566 INFO:teuthology.orchestra.run.trial039.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:09:32.615 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:09:32.616 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:09:32.629 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:09:32.631 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T08:09:32.632 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:09:32.659 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:09:32.662 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T08:09:32.669 INFO:teuthology.orchestra.run.trial039.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T08:09:32.673 INFO:teuthology.orchestra.run.trial039.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T08:09:32.676 INFO:teuthology.orchestra.run.trial039.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T08:09:32.680 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T08:09:32.692 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T08:09:32.705 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:09:32.706 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:09:32.727 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:09:32.730 INFO:teuthology.orchestra.run.trial039.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T08:09:32.762 INFO:teuthology.orchestra.run.trial039.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T08:09:32.767 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T08:09:32.774 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T08:09:32.776 INFO:teuthology.orchestra.run.trial039.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T08:09:32.780 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T08:09:32.897 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:09:32.898 INFO:teuthology.orchestra.run.trial039.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:09:32.924 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:09:32.926 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T08:09:32.932 INFO:teuthology.orchestra.run.trial039.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T08:09:33.186 INFO:teuthology.orchestra.run.trial039.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T08:09:33.187 INFO:teuthology.orchestra.run.trial039.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:09:33.203 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:09:33.208 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T08:09:33.212 INFO:teuthology.orchestra.run.trial039.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T08:09:33.215 INFO:teuthology.orchestra.run.trial039.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T08:09:33.215 INFO:teuthology.orchestra.run.trial039.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:09:33.232 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:09:33.344 INFO:teuthology.orchestra.run.trial039.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T08:09:33.346 INFO:teuthology.orchestra.run.trial039.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:09:33.625 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:09:33.625 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T08:09:33.626 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:09:33.669 INFO:teuthology.orchestra.run.trial039.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T08:09:33.671 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T08:09:34.634 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:09:34.636 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:09:34.901 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:09:34.904 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:09:34.945 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:09:34.945 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T08:09:34.945 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T08:09:34.945 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:09:34.948 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:09:39.410 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:09:39.411 INFO:teuthology.orchestra.run.trial039.stdout:skipping the directory /sys 2026-02-25T08:09:39.411 INFO:teuthology.orchestra.run.trial039.stdout:skipping the directory /proc 2026-02-25T08:09:39.411 INFO:teuthology.orchestra.run.trial039.stdout:skipping the directory /mnt 2026-02-25T08:09:39.411 INFO:teuthology.orchestra.run.trial039.stdout:skipping the directory /var/tmp 2026-02-25T08:09:39.411 INFO:teuthology.orchestra.run.trial039.stdout:skipping the directory /home 2026-02-25T08:09:39.411 INFO:teuthology.orchestra.run.trial039.stdout:skipping the directory /root 2026-02-25T08:09:39.411 INFO:teuthology.orchestra.run.trial039.stdout:skipping the directory /tmp 2026-02-25T08:09:39.411 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:09:39.916 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:09:39.939 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:09:39.939 INFO:teuthology.orchestra.run.trial039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:09:39.939 INFO:teuthology.orchestra.run.trial039.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T08:09:39.939 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:09:39.939 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:09:39.940 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:09:39.941 INFO:teuthology.orchestra.run.trial039.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:09:39.954 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:09:39.954 INFO:teuthology.orchestra.run.trial039.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:09:40.009 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:09:40.009 INFO:teuthology.orchestra.run.trial039.stdout:Creating group 'qat' with GID 992. 2026-02-25T08:09:40.009 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:09:40.303 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T08:09:40.303 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:09:40.303 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:09:40.303 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T08:09:40.303 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:09:40.303 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T08:09:40.303 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T08:09:40.303 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T08:09:40.304 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T08:09:40.305 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T08:09:40.306 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T08:09:40.306 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T08:09:40.306 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T08:09:40.306 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:09:40.306 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T08:09:40.306 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T08:09:40.372 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:09:40.372 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:09:40.372 INFO:teuthology.orchestra.run.trial039.stdout:Upgraded: 2026-02-25T08:09:40.372 INFO:teuthology.orchestra.run.trial039.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.372 INFO:teuthology.orchestra.run.trial039.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.372 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:09:40.372 INFO:teuthology.orchestra.run.trial039.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T08:09:40.372 INFO:teuthology.orchestra.run.trial039.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.372 INFO:teuthology.orchestra.run.trial039.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T08:09:40.373 INFO:teuthology.orchestra.run.trial039.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.374 INFO:teuthology.orchestra.run.trial039.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.375 INFO:teuthology.orchestra.run.trial039.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.375 INFO:teuthology.orchestra.run.trial039.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:09:40.375 INFO:teuthology.orchestra.run.trial039.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T08:09:40.375 INFO:teuthology.orchestra.run.trial039.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T08:09:40.375 INFO:teuthology.orchestra.run.trial039.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T08:09:40.375 INFO:teuthology.orchestra.run.trial039.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T08:09:40.375 INFO:teuthology.orchestra.run.trial039.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T08:09:40.375 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:09:40.375 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:09:40.412 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-test 2026-02-25T08:09:40.733 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:59 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:09:40.818 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:09:40.818 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:09:40.818 INFO:teuthology.orchestra.run.trial039.stdout: Package Arch Version Repository Size 2026-02-25T08:09:40.818 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout:Installing dependencies: 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T08:09:40.819 INFO:teuthology.orchestra.run.trial039.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T08:09:40.820 INFO:teuthology.orchestra.run.trial039.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T08:09:40.820 INFO:teuthology.orchestra.run.trial039.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T08:09:40.820 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:09:40.820 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:09:40.820 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:09:40.820 INFO:teuthology.orchestra.run.trial039.stdout:Install 12 Packages 2026-02-25T08:09:40.820 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:09:40.821 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 132 M 2026-02-25T08:09:40.821 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 988 M 2026-02-25T08:09:40.821 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:09:41.283 INFO:teuthology.orchestra.run.trial039.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 34 MB/s | 2.1 MB 00:00 2026-02-25T08:09:41.305 INFO:teuthology.orchestra.run.trial039.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 1.0 MB/s | 23 kB 00:00 2026-02-25T08:09:41.330 INFO:teuthology.orchestra.run.trial039.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 3.7 MB/s | 94 kB 00:00 2026-02-25T08:09:41.484 INFO:teuthology.orchestra.run.trial039.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 3.3 MB/s | 525 kB 00:00 2026-02-25T08:09:41.509 INFO:teuthology.orchestra.run.trial039.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 6.9 MB/s | 176 kB 00:00 2026-02-25T08:09:41.533 INFO:teuthology.orchestra.run.trial039.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 9.2 MB/s | 233 kB 00:00 2026-02-25T08:09:41.597 INFO:teuthology.orchestra.run.trial039.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 16 MB/s | 1.0 MB 00:00 2026-02-25T08:09:41.623 INFO:teuthology.orchestra.run.trial039.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 12 MB/s | 303 kB 00:00 2026-02-25T08:09:41.646 INFO:teuthology.orchestra.run.trial039.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.7 MB/s | 64 kB 00:00 2026-02-25T08:09:41.813 INFO:teuthology.orchestra.run.trial039.stdout:(10/12): libcephsqlite-20.3.0-5338.g6aeb6975.el 259 kB/s | 153 kB 00:00 2026-02-25T08:09:42.015 INFO:teuthology.orchestra.run.trial039.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 330 kB/s | 122 kB 00:00 2026-02-25T08:09:53.300 INFO:teuthology.orchestra.run.trial091.stdout:(27/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 297 kB/s | 21 MB 01:11 2026-02-25T08:11:17.873 INFO:teuthology.orchestra.run.trial091.stdout:[MIRROR] gperftools-libs-2.9.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://cfhcable.mm.fcix.net/epel/9/Everything/x86_64/Packages/g/gperftools-libs-2.9.1-3.el9.x86_64.rpm [Failed to connect to cfhcable.mm.fcix.net port 443: Connection timed out] 2026-02-25T08:11:17.873 INFO:teuthology.orchestra.run.trial091.stdout:[MIRROR] libarrow-9.0.0-15.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://cfhcable.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-15.el9.x86_64.rpm [Failed to connect to cfhcable.mm.fcix.net port 443: Connection timed out] 2026-02-25T08:11:18.125 INFO:teuthology.orchestra.run.trial091.stdout:(28/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.4 kB/s | 308 kB 02:10 2026-02-25T08:11:18.163 INFO:teuthology.orchestra.run.trial091.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 1.3 MB/s | 49 kB 00:00 2026-02-25T08:11:18.202 INFO:teuthology.orchestra.run.trial091.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 1.7 MB/s | 67 kB 00:00 2026-02-25T08:11:18.262 INFO:teuthology.orchestra.run.trial091.stdout:(31/37): libarrow-9.0.0-15.el9.x86_64.rpm 35 kB/s | 4.4 MB 02:09 2026-02-25T08:11:18.300 INFO:teuthology.orchestra.run.trial091.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 5.0 MB/s | 191 kB 00:00 2026-02-25T08:11:18.302 INFO:teuthology.orchestra.run.trial091.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2026-02-25T08:11:18.353 INFO:teuthology.orchestra.run.trial091.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 30 MB/s | 1.6 MB 00:00 2026-02-25T08:11:24.203 INFO:teuthology.orchestra.run.trial091.stdout:(35/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 570 kB/s | 3.3 MB 00:05 2026-02-25T08:11:26.507 INFO:teuthology.orchestra.run.trial091.stdout:(36/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 338 kB/s | 2.7 MB 00:08 2026-02-25T08:12:03.022 INFO:teuthology.orchestra.run.trial091.stdout:[MIRROR] libarrow-doc-9.0.0-15.el9.noarch.rpm: Curl error (28): Timeout was reached for https://cfhcable.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-15.el9.noarch.rpm [Failed to connect to cfhcable.mm.fcix.net port 443: Connection timed out] 2026-02-25T08:12:03.132 INFO:teuthology.orchestra.run.trial091.stdout:(37/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 195 B/s | 25 kB 02:09 2026-02-25T08:12:03.134 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:12:03.134 INFO:teuthology.orchestra.run.trial091.stdout:Total 311 kB/s | 61 MB 03:22 2026-02-25T08:12:03.220 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:12:03.240 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:12:03.240 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:12:03.486 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:12:03.487 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:12:03.780 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:12:03.921 INFO:teuthology.orchestra.run.trial091.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T08:12:03.925 INFO:teuthology.orchestra.run.trial091.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:12:03.974 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:12:03.975 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:12:03.987 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:12:03.989 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T08:12:03.990 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:12:04.017 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:12:04.019 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T08:12:04.027 INFO:teuthology.orchestra.run.trial091.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T08:12:04.029 INFO:teuthology.orchestra.run.trial091.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T08:12:04.033 INFO:teuthology.orchestra.run.trial091.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T08:12:04.037 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T08:12:04.048 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T08:12:04.061 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:12:04.062 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:12:04.085 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:12:04.089 INFO:teuthology.orchestra.run.trial091.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T08:12:04.122 INFO:teuthology.orchestra.run.trial091.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T08:12:04.126 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T08:12:04.132 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T08:12:04.134 INFO:teuthology.orchestra.run.trial091.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T08:12:04.139 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T08:12:04.252 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:12:04.253 INFO:teuthology.orchestra.run.trial091.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:12:04.279 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:12:04.281 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T08:12:04.287 INFO:teuthology.orchestra.run.trial091.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T08:12:04.535 INFO:teuthology.orchestra.run.trial091.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T08:12:04.536 INFO:teuthology.orchestra.run.trial091.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:12:04.552 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:12:04.555 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T08:12:04.560 INFO:teuthology.orchestra.run.trial091.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T08:12:04.562 INFO:teuthology.orchestra.run.trial091.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T08:12:04.563 INFO:teuthology.orchestra.run.trial091.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:12:04.580 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:12:04.695 INFO:teuthology.orchestra.run.trial091.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T08:12:04.698 INFO:teuthology.orchestra.run.trial091.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:12:04.986 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:12:04.986 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T08:12:04.986 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:12:05.029 INFO:teuthology.orchestra.run.trial091.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T08:12:05.031 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T08:12:05.969 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:12:05.972 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:12:06.233 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:12:06.237 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:12:06.272 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:12:06.273 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T08:12:06.273 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T08:12:06.273 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:12:06.276 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:12:10.770 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:12:10.770 INFO:teuthology.orchestra.run.trial091.stdout:skipping the directory /sys 2026-02-25T08:12:10.770 INFO:teuthology.orchestra.run.trial091.stdout:skipping the directory /proc 2026-02-25T08:12:10.770 INFO:teuthology.orchestra.run.trial091.stdout:skipping the directory /mnt 2026-02-25T08:12:10.771 INFO:teuthology.orchestra.run.trial091.stdout:skipping the directory /var/tmp 2026-02-25T08:12:10.771 INFO:teuthology.orchestra.run.trial091.stdout:skipping the directory /home 2026-02-25T08:12:10.771 INFO:teuthology.orchestra.run.trial091.stdout:skipping the directory /root 2026-02-25T08:12:10.771 INFO:teuthology.orchestra.run.trial091.stdout:skipping the directory /tmp 2026-02-25T08:12:10.771 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:12:11.261 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:12:11.284 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:12:11.284 INFO:teuthology.orchestra.run.trial091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:12:11.284 INFO:teuthology.orchestra.run.trial091.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T08:12:11.284 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:12:11.284 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:12:11.284 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:12:11.286 INFO:teuthology.orchestra.run.trial091.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:12:11.297 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:12:11.298 INFO:teuthology.orchestra.run.trial091.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:12:11.349 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:12:11.349 INFO:teuthology.orchestra.run.trial091.stdout:Creating group 'qat' with GID 992. 2026-02-25T08:12:11.349 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:12:11.636 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T08:12:11.636 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:12:11.636 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T08:12:11.637 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T08:12:11.638 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T08:12:11.639 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T08:12:11.703 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout:Upgraded: 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.704 INFO:teuthology.orchestra.run.trial091.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T08:12:11.705 INFO:teuthology.orchestra.run.trial091.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.706 INFO:teuthology.orchestra.run.trial091.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:11.707 INFO:teuthology.orchestra.run.trial091.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T08:12:11.707 INFO:teuthology.orchestra.run.trial091.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T08:12:11.707 INFO:teuthology.orchestra.run.trial091.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T08:12:11.707 INFO:teuthology.orchestra.run.trial091.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T08:12:11.707 INFO:teuthology.orchestra.run.trial091.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T08:12:11.707 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:12:11.707 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:12:11.745 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-test 2026-02-25T08:12:12.068 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:03:33 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:12:12.153 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: Package Arch Version Repository Size 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout:Installing dependencies: 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T08:12:12.154 INFO:teuthology.orchestra.run.trial091.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T08:12:12.155 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:12:12.155 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:12:12.155 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:12:12.155 INFO:teuthology.orchestra.run.trial091.stdout:Install 12 Packages 2026-02-25T08:12:12.155 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:12:12.155 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 132 M 2026-02-25T08:12:12.155 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 988 M 2026-02-25T08:12:12.155 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:12:13.152 INFO:teuthology.orchestra.run.trial091.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 8.9 MB/s | 2.1 MB 00:00 2026-02-25T08:12:13.177 INFO:teuthology.orchestra.run.trial091.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 905 kB/s | 23 kB 00:00 2026-02-25T08:12:13.203 INFO:teuthology.orchestra.run.trial091.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 3.5 MB/s | 94 kB 00:00 2026-02-25T08:12:13.544 INFO:teuthology.orchestra.run.trial091.stdout:(4/12): libcephsqlite-20.3.0-5338.g6aeb6975.el9 243 kB/s | 153 kB 00:00 2026-02-25T08:12:14.281 INFO:teuthology.orchestra.run.trial091.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 239 kB/s | 176 kB 00:00 2026-02-25T08:12:14.570 INFO:teuthology.orchestra.run.trial091.stdout:(6/12): libasan8-15.2.1-7.el9.x86_64.rpm 384 kB/s | 525 kB 00:01 2026-02-25T08:12:14.690 INFO:teuthology.orchestra.run.trial091.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 571 kB/s | 233 kB 00:00 2026-02-25T08:12:15.177 INFO:teuthology.orchestra.run.trial091.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 622 kB/s | 303 kB 00:00 2026-02-25T08:12:15.342 INFO:teuthology.orchestra.run.trial091.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 386 kB/s | 64 kB 00:00 2026-02-25T08:12:15.400 INFO:teuthology.orchestra.run.trial091.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.1 MB/s | 122 kB 00:00 2026-02-25T08:12:15.880 INFO:teuthology.orchestra.run.trial091.stdout:(11/12): protobuf-3.14.0-17.el9.x86_64.rpm 785 kB/s | 1.0 MB 00:01 2026-02-25T08:14:39.573 INFO:teuthology.orchestra.run.trial039.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 436 kB/s | 127 MB 04:58 2026-02-25T08:14:39.574 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:14:39.574 INFO:teuthology.orchestra.run.trial039.stdout:Total 451 kB/s | 132 MB 04:58 2026-02-25T08:14:39.647 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:14:39.658 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:14:39.658 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:14:40.091 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:14:40.091 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:14:40.550 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:14:40.559 INFO:teuthology.orchestra.run.trial039.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T08:14:40.567 INFO:teuthology.orchestra.run.trial039.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T08:14:40.573 INFO:teuthology.orchestra.run.trial039.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T08:14:40.577 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T08:14:40.579 INFO:teuthology.orchestra.run.trial039.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T08:14:40.584 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T08:14:40.585 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:14:40.600 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:14:40.602 INFO:teuthology.orchestra.run.trial039.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T08:14:40.612 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T08:14:40.620 INFO:teuthology.orchestra.run.trial039.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T08:14:40.621 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:14:47.565 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:14:47.584 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:14:48.354 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:14:48.355 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T08:14:48.355 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T08:14:48.355 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T08:14:48.355 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T08:14:48.355 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T08:14:48.355 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T08:14:48.355 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T08:14:48.355 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T08:14:48.355 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T08:14:48.356 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T08:14:48.356 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T08:14:48.414 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T08:14:48.414 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:14:48.414 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:14:48.414 INFO:teuthology.orchestra.run.trial039.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T08:14:48.415 INFO:teuthology.orchestra.run.trial039.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T08:14:48.416 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:14:48.416 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:14:48.455 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph 2026-02-25T08:14:48.774 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:06:07 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:14:48.861 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:14:48.864 INFO:teuthology.orchestra.run.trial039.stdout:======================================================================================= 2026-02-25T08:14:48.864 INFO:teuthology.orchestra.run.trial039.stdout: Package Arch Version Repository Size 2026-02-25T08:14:48.864 INFO:teuthology.orchestra.run.trial039.stdout:======================================================================================= 2026-02-25T08:14:48.864 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout:Installing dependencies: 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T08:14:48.865 INFO:teuthology.orchestra.run.trial039.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T08:14:48.866 INFO:teuthology.orchestra.run.trial039.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T08:14:48.867 INFO:teuthology.orchestra.run.trial039.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T08:14:48.868 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T08:14:48.869 INFO:teuthology.orchestra.run.trial039.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T08:14:48.870 INFO:teuthology.orchestra.run.trial039.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T08:14:48.871 INFO:teuthology.orchestra.run.trial039.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout:Installing weak dependencies: 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T08:14:48.872 INFO:teuthology.orchestra.run.trial039.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T08:14:48.873 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:14:48.873 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:14:48.873 INFO:teuthology.orchestra.run.trial039.stdout:======================================================================================= 2026-02-25T08:14:48.873 INFO:teuthology.orchestra.run.trial039.stdout:Install 96 Packages 2026-02-25T08:14:48.873 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:14:48.874 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 226 M 2026-02-25T08:14:48.874 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 1.5 G 2026-02-25T08:14:48.874 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:14:50.173 INFO:teuthology.orchestra.run.trial039.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 15 kB/s | 6.5 kB 00:00 2026-02-25T08:14:51.492 INFO:teuthology.orchestra.run.trial039.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 531 kB/s | 937 kB 00:01 2026-02-25T08:14:51.814 INFO:teuthology.orchestra.run.trial039.stdout:(3/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 507 kB/s | 163 kB 00:00 2026-02-25T08:14:54.368 INFO:teuthology.orchestra.run.trial039.stdout:(4/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 446 kB/s | 2.0 MB 00:04 2026-02-25T08:15:13.906 INFO:teuthology.orchestra.run.trial039.stdout:(5/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 210 kB/s | 4.9 MB 00:23 2026-02-25T08:15:14.046 INFO:teuthology.orchestra.run.trial039.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 410 kB/s | 57 kB 00:00 2026-02-25T08:15:14.467 INFO:teuthology.orchestra.run.trial039.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 437 kB/s | 184 kB 00:00 2026-02-25T08:15:29.582 INFO:teuthology.orchestra.run.trial039.stdout:(8/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 400 kB/s | 15 MB 00:37 2026-02-25T08:15:49.361 INFO:teuthology.orchestra.run.trial039.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 384 kB/s | 7.4 MB 00:19 2026-02-25T08:15:49.430 INFO:teuthology.orchestra.run.trial039.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 322 kB/s | 22 kB 00:00 2026-02-25T08:15:50.202 INFO:teuthology.orchestra.run.trial039.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 384 kB/s | 297 kB 00:00 2026-02-25T08:15:50.572 INFO:teuthology.orchestra.run.trial039.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 136 kB/s | 50 kB 00:00 2026-02-25T08:15:50.711 INFO:teuthology.orchestra.run.trial039.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 125 kB/s | 17 kB 00:00 2026-02-25T08:15:52.181 INFO:teuthology.orchestra.run.trial039.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 205 kB/s | 301 kB 00:01 2026-02-25T08:15:57.143 INFO:teuthology.orchestra.run.trial039.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 210 kB/s | 1.0 MB 00:04 2026-02-25T08:15:57.250 INFO:teuthology.orchestra.run.trial039.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 3.2 MB/s | 351 kB 00:00 2026-02-25T08:15:57.272 INFO:teuthology.orchestra.run.trial039.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.8 MB/s | 40 kB 00:00 2026-02-25T08:15:57.294 INFO:teuthology.orchestra.run.trial039.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.2 MB/s | 72 kB 00:00 2026-02-25T08:15:57.338 INFO:teuthology.orchestra.run.trial039.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 18 MB/s | 794 kB 00:00 2026-02-25T08:15:57.363 INFO:teuthology.orchestra.run.trial039.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 7.2 MB/s | 184 kB 00:00 2026-02-25T08:15:57.388 INFO:teuthology.orchestra.run.trial039.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 10 MB/s | 253 kB 00:00 2026-02-25T08:15:57.475 INFO:teuthology.orchestra.run.trial039.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 14 MB/s | 1.2 MB 00:00 2026-02-25T08:15:57.499 INFO:teuthology.orchestra.run.trial039.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.3 MB/s | 106 kB 00:00 2026-02-25T08:15:57.523 INFO:teuthology.orchestra.run.trial039.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 5.5 MB/s | 135 kB 00:00 2026-02-25T08:15:57.547 INFO:teuthology.orchestra.run.trial039.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.0 MB/s | 150 kB 00:00 2026-02-25T08:15:57.573 INFO:teuthology.orchestra.run.trial039.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 5.0 MB/s | 126 kB 00:00 2026-02-25T08:15:57.597 INFO:teuthology.orchestra.run.trial039.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 8.6 MB/s | 218 kB 00:00 2026-02-25T08:15:57.622 INFO:teuthology.orchestra.run.trial039.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 7.1 MB/s | 182 kB 00:00 2026-02-25T08:15:57.646 INFO:teuthology.orchestra.run.trial039.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-25T08:15:57.737 INFO:teuthology.orchestra.run.trial039.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 328 kB/s | 30 kB 00:00 2026-02-25T08:15:57.926 INFO:teuthology.orchestra.run.trial039.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 16 MB/s | 3.0 MB 00:00 2026-02-25T08:15:57.951 INFO:teuthology.orchestra.run.trial039.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 587 kB/s | 15 kB 00:00 2026-02-25T08:15:57.978 INFO:teuthology.orchestra.run.trial039.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 8.9 MB/s | 246 kB 00:00 2026-02-25T08:15:58.005 INFO:teuthology.orchestra.run.trial039.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 6.8 MB/s | 188 kB 00:00 2026-02-25T08:15:58.031 INFO:teuthology.orchestra.run.trial039.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 1.6 MB/s | 42 kB 00:00 2026-02-25T08:15:58.097 INFO:teuthology.orchestra.run.trial039.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 81 MB/s | 5.3 MB 00:00 2026-02-25T08:15:58.162 INFO:teuthology.orchestra.run.trial039.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 92 MB/s | 6.0 MB 00:00 2026-02-25T08:15:58.190 INFO:teuthology.orchestra.run.trial039.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 8.4 MB/s | 244 kB 00:00 2026-02-25T08:15:58.218 INFO:teuthology.orchestra.run.trial039.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 8.7 MB/s | 249 kB 00:00 2026-02-25T08:15:58.244 INFO:teuthology.orchestra.run.trial039.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 1.8 MB/s | 48 kB 00:00 2026-02-25T08:15:58.271 INFO:teuthology.orchestra.run.trial039.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 6.5 MB/s | 177 kB 00:00 2026-02-25T08:15:58.302 INFO:teuthology.orchestra.run.trial039.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 40 MB/s | 1.2 MB 00:00 2026-02-25T08:15:58.327 INFO:teuthology.orchestra.run.trial039.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 1.3 MB/s | 35 kB 00:00 2026-02-25T08:15:58.393 INFO:teuthology.orchestra.run.trial039.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 94 MB/s | 6.1 MB 00:00 2026-02-25T08:15:58.421 INFO:teuthology.orchestra.run.trial039.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 15 MB/s | 442 kB 00:00 2026-02-25T08:15:58.448 INFO:teuthology.orchestra.run.trial039.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 2.9 MB/s | 77 kB 00:00 2026-02-25T08:15:58.475 INFO:teuthology.orchestra.run.trial039.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 9.7 MB/s | 267 kB 00:00 2026-02-25T08:15:58.502 INFO:teuthology.orchestra.run.trial039.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 5.6 MB/s | 157 kB 00:00 2026-02-25T08:15:58.530 INFO:teuthology.orchestra.run.trial039.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 9.7 MB/s | 277 kB 00:00 2026-02-25T08:15:58.556 INFO:teuthology.orchestra.run.trial039.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 2.0 MB/s | 54 kB 00:00 2026-02-25T08:15:58.730 INFO:teuthology.orchestra.run.trial039.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 111 MB/s | 19 MB 00:00 2026-02-25T08:15:58.756 INFO:teuthology.orchestra.run.trial039.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 42 kB 00:00 2026-02-25T08:15:58.783 INFO:teuthology.orchestra.run.trial039.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 7.0 MB/s | 189 kB 00:00 2026-02-25T08:15:58.809 INFO:teuthology.orchestra.run.trial039.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 3.4 MB/s | 90 kB 00:00 2026-02-25T08:15:59.355 INFO:teuthology.orchestra.run.trial039.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 41 kB/s | 22 kB 00:00 2026-02-25T08:16:00.252 INFO:teuthology.orchestra.run.trial039.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 960 kB/s | 862 kB 00:00 2026-02-25T08:16:00.530 INFO:teuthology.orchestra.run.trial039.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.9 MB/s | 551 kB 00:00 2026-02-25T08:16:00.566 INFO:teuthology.orchestra.run.trial039.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 544 kB/s | 19 kB 00:00 2026-02-25T08:16:00.607 INFO:teuthology.orchestra.run.trial039.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2026-02-25T08:16:00.666 INFO:teuthology.orchestra.run.trial039.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 9.0 MB/s | 548 kB 00:00 2026-02-25T08:16:00.703 INFO:teuthology.orchestra.run.trial039.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 813 kB/s | 29 kB 00:00 2026-02-25T08:16:00.740 INFO:teuthology.orchestra.run.trial039.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 1.6 MB/s | 60 kB 00:00 2026-02-25T08:16:00.777 INFO:teuthology.orchestra.run.trial039.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.2 MB/s | 43 kB 00:00 2026-02-25T08:16:00.813 INFO:teuthology.orchestra.run.trial039.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 886 kB/s | 32 kB 00:00 2026-02-25T08:16:00.849 INFO:teuthology.orchestra.run.trial039.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 392 kB/s | 14 kB 00:00 2026-02-25T08:16:00.891 INFO:teuthology.orchestra.run.trial039.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 4.1 MB/s | 173 kB 00:00 2026-02-25T08:16:00.939 INFO:teuthology.orchestra.run.trial039.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 7.2 MB/s | 358 kB 00:00 2026-02-25T08:16:00.983 INFO:teuthology.orchestra.run.trial039.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 5.6 MB/s | 254 kB 00:00 2026-02-25T08:16:01.092 INFO:teuthology.orchestra.run.trial039.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 19 MB/s | 2.0 MB 00:00 2026-02-25T08:16:01.130 INFO:teuthology.orchestra.run.trial039.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 3.7 MB/s | 144 kB 00:00 2026-02-25T08:16:01.169 INFO:teuthology.orchestra.run.trial039.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 3.5 MB/s | 139 kB 00:00 2026-02-25T08:16:01.205 INFO:teuthology.orchestra.run.trial039.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.5 MB/s | 56 kB 00:00 2026-02-25T08:16:01.242 INFO:teuthology.orchestra.run.trial039.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 296 kB/s | 11 kB 00:00 2026-02-25T08:16:01.278 INFO:teuthology.orchestra.run.trial039.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 488 kB/s | 18 kB 00:00 2026-02-25T08:16:01.314 INFO:teuthology.orchestra.run.trial039.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 640 kB/s | 23 kB 00:00 2026-02-25T08:16:01.350 INFO:teuthology.orchestra.run.trial039.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 545 kB/s | 20 kB 00:00 2026-02-25T08:16:01.387 INFO:teuthology.orchestra.run.trial039.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 539 kB/s | 19 kB 00:00 2026-02-25T08:16:01.423 INFO:teuthology.orchestra.run.trial039.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 728 kB/s | 26 kB 00:00 2026-02-25T08:16:01.471 INFO:teuthology.orchestra.run.trial039.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 21 MB/s | 1.0 MB 00:00 2026-02-25T08:16:01.508 INFO:teuthology.orchestra.run.trial039.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 2.1 MB/s | 79 kB 00:00 2026-02-25T08:16:01.545 INFO:teuthology.orchestra.run.trial039.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.3 MB/s | 86 kB 00:00 2026-02-25T08:16:01.582 INFO:teuthology.orchestra.run.trial039.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.5 MB/s | 58 kB 00:00 2026-02-25T08:16:01.618 INFO:teuthology.orchestra.run.trial039.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 457 kB/s | 16 kB 00:00 2026-02-25T08:16:01.655 INFO:teuthology.orchestra.run.trial039.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.4 MB/s | 90 kB 00:00 2026-02-25T08:16:01.691 INFO:teuthology.orchestra.run.trial039.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 849 kB/s | 31 kB 00:00 2026-02-25T08:16:01.729 INFO:teuthology.orchestra.run.trial039.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 4.9 MB/s | 188 kB 00:00 2026-02-25T08:16:01.765 INFO:teuthology.orchestra.run.trial039.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 1.6 MB/s | 59 kB 00:00 2026-02-25T08:16:01.803 INFO:teuthology.orchestra.run.trial039.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 3.3 MB/s | 125 kB 00:00 2026-02-25T08:16:01.839 INFO:teuthology.orchestra.run.trial039.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 981 kB/s | 36 kB 00:00 2026-02-25T08:16:01.876 INFO:teuthology.orchestra.run.trial039.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 2.3 MB/s | 86 kB 00:00 2026-02-25T08:16:01.912 INFO:teuthology.orchestra.run.trial039.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 2.4 MB/s | 90 kB 00:00 2026-02-25T08:16:01.949 INFO:teuthology.orchestra.run.trial039.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.3 MB/s | 48 kB 00:00 2026-02-25T08:16:01.985 INFO:teuthology.orchestra.run.trial039.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 614 kB/s | 22 kB 00:00 2026-02-25T08:16:02.022 INFO:teuthology.orchestra.run.trial039.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 553 kB/s | 20 kB 00:00 2026-02-25T08:17:35.139 INFO:teuthology.orchestra.run.trial091.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 403 kB/s | 127 MB 05:22 2026-02-25T08:17:35.142 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:17:35.142 INFO:teuthology.orchestra.run.trial091.stdout:Total 417 kB/s | 132 MB 05:22 2026-02-25T08:17:35.214 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:17:35.225 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:17:35.225 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:17:35.656 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:17:35.656 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:17:36.113 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:17:36.121 INFO:teuthology.orchestra.run.trial091.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T08:17:36.130 INFO:teuthology.orchestra.run.trial091.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T08:17:36.136 INFO:teuthology.orchestra.run.trial091.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T08:17:36.139 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T08:17:36.142 INFO:teuthology.orchestra.run.trial091.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T08:17:36.146 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T08:17:36.147 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:17:36.162 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:17:36.164 INFO:teuthology.orchestra.run.trial091.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T08:17:36.174 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T08:17:36.183 INFO:teuthology.orchestra.run.trial091.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T08:17:36.183 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:17:43.131 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:17:43.150 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:17:43.946 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:17:43.946 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T08:17:43.946 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T08:17:43.946 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T08:17:43.946 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T08:17:43.946 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T08:17:43.947 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T08:17:43.947 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T08:17:43.947 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T08:17:43.947 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T08:17:43.947 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T08:17:43.947 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T08:17:44.005 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T08:17:44.005 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:17:44.005 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:17:44.005 INFO:teuthology.orchestra.run.trial091.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:17:44.005 INFO:teuthology.orchestra.run.trial091.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T08:17:44.005 INFO:teuthology.orchestra.run.trial091.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T08:17:44.005 INFO:teuthology.orchestra.run.trial091.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T08:17:44.005 INFO:teuthology.orchestra.run.trial091.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:17:44.006 INFO:teuthology.orchestra.run.trial091.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T08:17:44.006 INFO:teuthology.orchestra.run.trial091.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T08:17:44.006 INFO:teuthology.orchestra.run.trial091.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T08:17:44.006 INFO:teuthology.orchestra.run.trial091.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T08:17:44.006 INFO:teuthology.orchestra.run.trial091.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T08:17:44.006 INFO:teuthology.orchestra.run.trial091.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T08:17:44.006 INFO:teuthology.orchestra.run.trial091.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T08:17:44.006 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:17:44.006 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:17:44.045 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph 2026-02-25T08:17:44.367 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:09:05 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:17:44.456 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:17:44.458 INFO:teuthology.orchestra.run.trial091.stdout:======================================================================================= 2026-02-25T08:17:44.458 INFO:teuthology.orchestra.run.trial091.stdout: Package Arch Version Repository Size 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout:======================================================================================= 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout:Installing dependencies: 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T08:17:44.459 INFO:teuthology.orchestra.run.trial091.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T08:17:44.460 INFO:teuthology.orchestra.run.trial091.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T08:17:44.461 INFO:teuthology.orchestra.run.trial091.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T08:17:44.462 INFO:teuthology.orchestra.run.trial091.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout:Installing weak dependencies: 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout:======================================================================================= 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout:Install 96 Packages 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 226 M 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 1.5 G 2026-02-25T08:17:44.463 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:17:45.650 INFO:teuthology.orchestra.run.trial091.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-25T08:17:46.869 INFO:teuthology.orchestra.run.trial091.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 623 kB/s | 937 kB 00:01 2026-02-25T08:17:47.190 INFO:teuthology.orchestra.run.trial091.stdout:(3/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 508 kB/s | 163 kB 00:00 2026-02-25T08:17:47.214 INFO:teuthology.orchestra.run.trial091.stdout:(4/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 1.1 MB/s | 2.0 MB 00:01 2026-02-25T08:18:05.453 INFO:teuthology.orchestra.run.trial091.stdout:(5/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 252 kB/s | 4.9 MB 00:19 2026-02-25T08:18:06.033 INFO:teuthology.orchestra.run.trial091.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 99 kB/s | 57 kB 00:00 2026-02-25T08:18:07.404 INFO:teuthology.orchestra.run.trial091.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 134 kB/s | 184 kB 00:01 2026-02-25T08:18:51.766 INFO:teuthology.orchestra.run.trial091.stdout:(8/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 234 kB/s | 15 MB 01:04 2026-02-25T08:19:05.428 INFO:teuthology.orchestra.run.trial039.stdout:(95/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 358 kB/s | 88 MB 04:11 2026-02-25T08:19:09.644 INFO:teuthology.orchestra.run.trial039.stdout:(96/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 221 kB/s | 51 MB 03:55 2026-02-25T08:19:09.646 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:09.646 INFO:teuthology.orchestra.run.trial039.stdout:Total 886 kB/s | 226 MB 04:20 2026-02-25T08:19:10.011 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:19:10.049 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:19:10.049 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:19:10.850 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:19:10.851 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:19:11.678 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:19:11.680 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T08:19:11.683 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T08:19:11.705 INFO:teuthology.orchestra.run.trial039.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T08:19:11.738 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T08:19:11.744 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T08:19:11.747 INFO:teuthology.orchestra.run.trial039.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T08:19:11.751 INFO:teuthology.orchestra.run.trial039.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T08:19:11.774 INFO:teuthology.orchestra.run.trial039.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T08:19:11.787 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T08:19:11.791 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T08:19:11.797 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T08:19:11.799 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T08:19:11.810 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T08:19:11.816 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T08:19:11.841 INFO:teuthology.orchestra.run.trial039.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T08:19:11.844 INFO:teuthology.orchestra.run.trial039.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T08:19:11.894 INFO:teuthology.orchestra.run.trial039.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T08:19:11.907 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T08:19:11.909 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T08:19:11.914 INFO:teuthology.orchestra.run.trial039.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T08:19:11.919 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T08:19:11.924 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T08:19:11.927 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T08:19:11.942 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T08:19:11.946 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T08:19:11.951 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T08:19:11.962 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T08:19:11.972 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T08:19:11.978 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T08:19:11.997 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T08:19:12.009 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T08:19:12.024 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T08:19:12.029 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T08:19:12.037 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T08:19:12.077 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T08:19:12.404 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T08:19:12.414 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T08:19:12.417 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T08:19:12.423 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T08:19:12.425 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T08:19:12.453 INFO:teuthology.orchestra.run.trial039.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T08:19:12.497 INFO:teuthology.orchestra.run.trial039.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T08:19:12.505 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T08:19:12.510 INFO:teuthology.orchestra.run.trial039.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T08:19:12.514 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T08:19:12.520 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T08:19:12.523 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T08:19:12.530 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T08:19:12.534 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T08:19:12.562 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T08:19:12.571 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T08:19:12.577 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T08:19:12.584 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T08:19:12.619 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T08:19:12.844 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T08:19:12.868 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T08:19:12.872 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T08:19:12.930 INFO:teuthology.orchestra.run.trial039.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T08:19:12.932 INFO:teuthology.orchestra.run.trial039.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T08:19:12.953 INFO:teuthology.orchestra.run.trial039.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T08:19:13.281 INFO:teuthology.orchestra.run.trial039.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T08:19:13.354 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T08:19:14.030 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T08:19:14.047 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T08:19:14.057 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T08:19:14.064 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T08:19:14.077 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T08:19:14.093 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T08:19:14.169 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T08:19:14.178 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T08:19:14.202 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T08:19:14.232 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T08:19:14.283 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T08:19:14.289 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T08:19:14.293 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T08:19:14.294 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T08:19:14.312 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:19:14.313 INFO:teuthology.orchestra.run.trial039.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T08:19:14.313 INFO:teuthology.orchestra.run.trial039.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T08:19:14.313 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:14.322 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:19:14.346 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:19:14.346 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T08:19:14.346 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:14.382 INFO:teuthology.orchestra.run.trial039.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T08:19:14.973 INFO:teuthology.orchestra.run.trial039.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T08:19:20.415 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T08:19:20.479 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T08:19:20.482 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:19:20.495 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:19:20.496 INFO:teuthology.orchestra.run.trial039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:20.496 INFO:teuthology.orchestra.run.trial039.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T08:19:20.496 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:20.506 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:19:20.529 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:19:20.530 INFO:teuthology.orchestra.run.trial039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:20.530 INFO:teuthology.orchestra.run.trial039.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T08:19:20.530 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:19:20.530 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:19:20.530 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:20.588 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:19:20.590 INFO:teuthology.orchestra.run.trial039.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:19:20.599 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T08:19:20.623 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T08:19:20.624 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:19:23.203 INFO:teuthology.orchestra.run.trial091.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 241 kB/s | 7.4 MB 00:31 2026-02-25T08:19:23.436 INFO:teuthology.orchestra.run.trial091.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 96 kB/s | 22 kB 00:00 2026-02-25T08:19:23.535 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:19:23.542 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:19:23.999 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:19:24.000 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:19:24.011 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:19:24.012 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:19:24.074 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:19:24.114 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T08:19:24.115 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:19:24.136 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:19:24.136 INFO:teuthology.orchestra.run.trial039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:24.136 INFO:teuthology.orchestra.run.trial039.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T08:19:24.136 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:19:24.136 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:19:24.136 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:24.148 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:19:24.362 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:19:24.363 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:19:24.383 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:19:24.383 INFO:teuthology.orchestra.run.trial039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:24.383 INFO:teuthology.orchestra.run.trial039.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T08:19:24.383 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:19:24.384 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:19:24.384 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:24.467 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:19:24.487 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:19:24.487 INFO:teuthology.orchestra.run.trial039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:24.487 INFO:teuthology.orchestra.run.trial039.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T08:19:24.488 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:19:24.488 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:19:24.488 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:24.489 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:19:24.494 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:19:24.517 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:19:25.042 INFO:teuthology.orchestra.run.trial091.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 185 kB/s | 297 kB 00:01 2026-02-25T08:19:25.557 INFO:teuthology.orchestra.run.trial091.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 98 kB/s | 50 kB 00:00 2026-02-25T08:19:25.938 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:19:25.938 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T08:19:25.938 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T08:19:25.938 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T08:19:25.938 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T08:19:25.938 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T08:19:25.938 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T08:19:25.939 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T08:19:25.940 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T08:19:25.941 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T08:19:25.942 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T08:19:25.943 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T08:19:25.944 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T08:19:25.945 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T08:19:25.946 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T08:19:25.989 INFO:teuthology.orchestra.run.trial091.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 40 kB/s | 17 kB 00:00 2026-02-25T08:19:26.006 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T08:19:26.007 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:26.007 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:19:26.007 INFO:teuthology.orchestra.run.trial039.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T08:19:26.007 INFO:teuthology.orchestra.run.trial039.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:26.007 INFO:teuthology.orchestra.run.trial039.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:26.007 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:26.007 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:26.007 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:26.007 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:26.007 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T08:19:26.008 INFO:teuthology.orchestra.run.trial039.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T08:19:26.009 INFO:teuthology.orchestra.run.trial039.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T08:19:26.010 INFO:teuthology.orchestra.run.trial039.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T08:19:26.011 INFO:teuthology.orchestra.run.trial039.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T08:19:26.012 INFO:teuthology.orchestra.run.trial039.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T08:19:26.013 INFO:teuthology.orchestra.run.trial039.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T08:19:26.014 INFO:teuthology.orchestra.run.trial039.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T08:19:26.015 INFO:teuthology.orchestra.run.trial039.stdout: zip-3.0-35.el9.x86_64 2026-02-25T08:19:26.015 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:26.015 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:26.075 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-base 2026-02-25T08:19:26.399 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:45 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:26.465 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:26.483 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:26.484 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:26.484 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:26.506 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install cephadm 2026-02-25T08:19:26.826 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:45 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:26.891 INFO:teuthology.orchestra.run.trial039.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:26.910 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:26.911 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:26.911 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:26.931 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:19:27.247 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:46 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:27.332 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:27.332 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:27.332 INFO:teuthology.orchestra.run.trial039.stdout: Package Arch Version Repo Size 2026-02-25T08:19:27.332 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:27.332 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:19:27.332 INFO:teuthology.orchestra.run.trial039.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T08:19:27.333 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:27.333 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:19:27.333 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:27.333 INFO:teuthology.orchestra.run.trial039.stdout:Install 1 Package 2026-02-25T08:19:27.333 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:27.333 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 149 k 2026-02-25T08:19:27.333 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 455 k 2026-02-25T08:19:27.333 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:19:28.146 INFO:teuthology.orchestra.run.trial039.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 183 kB/s | 149 kB 00:00 2026-02-25T08:19:28.147 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:28.148 INFO:teuthology.orchestra.run.trial039.stdout:Total 183 kB/s | 149 kB 00:00 2026-02-25T08:19:28.148 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:19:28.152 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:19:28.152 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:19:28.184 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:19:28.184 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:19:28.240 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:19:28.241 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:19:28.263 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:19:28.263 INFO:teuthology.orchestra.run.trial039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:28.263 INFO:teuthology.orchestra.run.trial039.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T08:19:28.263 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:28.587 INFO:teuthology.orchestra.run.trial091.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 116 kB/s | 301 kB 00:02 2026-02-25T08:19:28.589 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:19:28.589 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:28.589 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:19:28.589 INFO:teuthology.orchestra.run.trial039.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:28.589 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:28.589 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:28.618 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-mgr 2026-02-25T08:19:28.942 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:47 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:29.008 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:29.027 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:29.028 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:29.028 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:29.049 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:19:29.369 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:48 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:29.433 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:29.451 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:29.452 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:29.452 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:29.473 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:19:29.791 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:48 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:29.857 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:29.875 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:29.876 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:29.876 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:29.897 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:19:30.211 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:49 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:30.276 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:30.295 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:30.296 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:30.296 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:30.317 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:19:30.635 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:49 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:30.699 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:30.718 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:30.719 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:30.719 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:30.740 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-osd 2026-02-25T08:19:31.062 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:50 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:31.130 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:31.148 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:31.149 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:31.149 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:31.170 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-osd-classic 2026-02-25T08:19:31.485 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:50 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:31.551 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:31.569 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:31.570 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:31.570 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:31.591 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-fuse 2026-02-25T08:19:31.920 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:50 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:32.004 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:32.005 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:32.005 INFO:teuthology.orchestra.run.trial039.stdout: Package Arch Version Repo Size 2026-02-25T08:19:32.005 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:32.005 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:19:32.005 INFO:teuthology.orchestra.run.trial039.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T08:19:32.006 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:32.006 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:19:32.006 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:32.006 INFO:teuthology.orchestra.run.trial039.stdout:Install 1 Package 2026-02-25T08:19:32.006 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:32.006 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 914 k 2026-02-25T08:19:32.006 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 2.9 M 2026-02-25T08:19:32.006 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:19:33.419 INFO:teuthology.orchestra.run.trial039.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 647 kB/s | 914 kB 00:01 2026-02-25T08:19:33.419 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:33.419 INFO:teuthology.orchestra.run.trial039.stdout:Total 646 kB/s | 914 kB 00:01 2026-02-25T08:19:33.419 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:19:33.424 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:19:33.424 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:19:33.457 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:19:33.458 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:19:33.549 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:19:33.567 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:33.826 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:33.890 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:33.899 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:33.900 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:19:33.900 INFO:teuthology.orchestra.run.trial039.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:33.900 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:33.900 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:33.918 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-volume 2026-02-25T08:19:34.234 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:53 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:34.299 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:19:34.318 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:34.319 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:34.319 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:34.340 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install librados-devel 2026-02-25T08:19:34.649 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:53 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:34.733 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:34.734 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:34.734 INFO:teuthology.orchestra.run.trial039.stdout: Package Arch Version Repo Size 2026-02-25T08:19:34.734 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:34.734 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:19:34.734 INFO:teuthology.orchestra.run.trial039.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T08:19:34.735 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:34.735 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:19:34.735 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:34.735 INFO:teuthology.orchestra.run.trial039.stdout:Install 1 Package 2026-02-25T08:19:34.735 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:34.735 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 124 k 2026-02-25T08:19:34.735 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 434 k 2026-02-25T08:19:34.735 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:19:35.369 INFO:teuthology.orchestra.run.trial039.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 195 kB/s | 124 kB 00:00 2026-02-25T08:19:35.369 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:35.369 INFO:teuthology.orchestra.run.trial039.stdout:Total 194 kB/s | 124 kB 00:00 2026-02-25T08:19:35.369 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:19:35.373 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:19:35.373 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:19:35.404 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:19:35.404 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:19:35.460 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:19:35.477 INFO:teuthology.orchestra.run.trial039.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:35.570 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:35.637 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:35.637 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:35.637 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:19:35.637 INFO:teuthology.orchestra.run.trial039.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:35.637 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:35.637 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:35.663 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install libcephfs2 2026-02-25T08:19:35.980 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:54 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:36.045 INFO:teuthology.orchestra.run.trial039.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:36.063 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:36.064 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:36.064 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:36.085 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install libcephfs-devel 2026-02-25T08:19:36.174 INFO:teuthology.orchestra.run.trial091.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 137 kB/s | 1.0 MB 00:07 2026-02-25T08:19:36.282 INFO:teuthology.orchestra.run.trial091.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 3.2 MB/s | 351 kB 00:00 2026-02-25T08:19:36.304 INFO:teuthology.orchestra.run.trial091.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.8 MB/s | 40 kB 00:00 2026-02-25T08:19:36.326 INFO:teuthology.orchestra.run.trial091.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.2 MB/s | 72 kB 00:00 2026-02-25T08:19:36.370 INFO:teuthology.orchestra.run.trial091.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 18 MB/s | 794 kB 00:00 2026-02-25T08:19:36.395 INFO:teuthology.orchestra.run.trial091.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 7.3 MB/s | 184 kB 00:00 2026-02-25T08:19:36.406 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:55 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:36.420 INFO:teuthology.orchestra.run.trial091.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 10 MB/s | 253 kB 00:00 2026-02-25T08:19:36.485 INFO:teuthology.orchestra.run.trial091.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 19 MB/s | 1.2 MB 00:00 2026-02-25T08:19:36.490 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:36.491 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:36.491 INFO:teuthology.orchestra.run.trial039.stdout: Package Arch Version Repo Size 2026-02-25T08:19:36.491 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:36.491 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:19:36.491 INFO:teuthology.orchestra.run.trial039.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:19:36.491 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:36.491 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:19:36.492 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:36.492 INFO:teuthology.orchestra.run.trial039.stdout:Install 1 Package 2026-02-25T08:19:36.492 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:36.492 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 37 k 2026-02-25T08:19:36.492 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 171 k 2026-02-25T08:19:36.492 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:19:36.509 INFO:teuthology.orchestra.run.trial091.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.4 MB/s | 106 kB 00:00 2026-02-25T08:19:36.533 INFO:teuthology.orchestra.run.trial091.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 5.6 MB/s | 135 kB 00:00 2026-02-25T08:19:36.556 INFO:teuthology.orchestra.run.trial091.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.2 MB/s | 150 kB 00:00 2026-02-25T08:19:36.581 INFO:teuthology.orchestra.run.trial091.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 5.1 MB/s | 126 kB 00:00 2026-02-25T08:19:36.605 INFO:teuthology.orchestra.run.trial091.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 8.7 MB/s | 218 kB 00:00 2026-02-25T08:19:36.630 INFO:teuthology.orchestra.run.trial091.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 7.2 MB/s | 182 kB 00:00 2026-02-25T08:19:36.654 INFO:teuthology.orchestra.run.trial091.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-25T08:19:36.728 INFO:teuthology.orchestra.run.trial091.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 397 kB/s | 30 kB 00:00 2026-02-25T08:19:36.887 INFO:teuthology.orchestra.run.trial091.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 19 MB/s | 3.0 MB 00:00 2026-02-25T08:19:36.910 INFO:teuthology.orchestra.run.trial091.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 649 kB/s | 15 kB 00:00 2026-02-25T08:19:36.935 INFO:teuthology.orchestra.run.trial091.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 9.7 MB/s | 246 kB 00:00 2026-02-25T08:19:36.959 INFO:teuthology.orchestra.run.trial091.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 7.5 MB/s | 188 kB 00:00 2026-02-25T08:19:36.982 INFO:teuthology.orchestra.run.trial091.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 1.8 MB/s | 42 kB 00:00 2026-02-25T08:19:37.050 INFO:teuthology.orchestra.run.trial091.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 79 MB/s | 5.3 MB 00:00 2026-02-25T08:19:37.111 INFO:teuthology.orchestra.run.trial091.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 98 MB/s | 6.0 MB 00:00 2026-02-25T08:19:37.136 INFO:teuthology.orchestra.run.trial091.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 9.4 MB/s | 244 kB 00:00 2026-02-25T08:19:37.162 INFO:teuthology.orchestra.run.trial091.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 9.6 MB/s | 249 kB 00:00 2026-02-25T08:19:37.185 INFO:teuthology.orchestra.run.trial091.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 2.0 MB/s | 48 kB 00:00 2026-02-25T08:19:37.209 INFO:teuthology.orchestra.run.trial091.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 7.1 MB/s | 177 kB 00:00 2026-02-25T08:19:37.238 INFO:teuthology.orchestra.run.trial091.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 42 MB/s | 1.2 MB 00:00 2026-02-25T08:19:37.262 INFO:teuthology.orchestra.run.trial091.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 1.5 MB/s | 35 kB 00:00 2026-02-25T08:19:37.342 INFO:teuthology.orchestra.run.trial091.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 77 MB/s | 6.1 MB 00:00 2026-02-25T08:19:37.368 INFO:teuthology.orchestra.run.trial091.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 16 MB/s | 442 kB 00:00 2026-02-25T08:19:37.392 INFO:teuthology.orchestra.run.trial091.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 3.1 MB/s | 77 kB 00:00 2026-02-25T08:19:37.417 INFO:teuthology.orchestra.run.trial091.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 11 MB/s | 267 kB 00:00 2026-02-25T08:19:37.442 INFO:teuthology.orchestra.run.trial091.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 6.2 MB/s | 157 kB 00:00 2026-02-25T08:19:37.457 INFO:teuthology.orchestra.run.trial039.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 39 kB/s | 37 kB 00:00 2026-02-25T08:19:37.458 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:37.458 INFO:teuthology.orchestra.run.trial039.stdout:Total 39 kB/s | 37 kB 00:00 2026-02-25T08:19:37.458 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:19:37.460 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:19:37.460 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:19:37.463 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:19:37.464 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:19:37.467 INFO:teuthology.orchestra.run.trial091.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 11 MB/s | 277 kB 00:00 2026-02-25T08:19:37.486 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:19:37.491 INFO:teuthology.orchestra.run.trial091.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 2.2 MB/s | 54 kB 00:00 2026-02-25T08:19:37.503 INFO:teuthology.orchestra.run.trial039.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:37.564 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:37.626 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:37.627 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:37.627 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:19:37.627 INFO:teuthology.orchestra.run.trial039.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:37.627 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:37.627 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:37.654 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install librados2 2026-02-25T08:19:37.809 INFO:teuthology.orchestra.run.trial091.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 61 MB/s | 19 MB 00:00 2026-02-25T08:19:37.856 INFO:teuthology.orchestra.run.trial091.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 893 kB/s | 42 kB 00:00 2026-02-25T08:19:37.880 INFO:teuthology.orchestra.run.trial091.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 7.7 MB/s | 189 kB 00:00 2026-02-25T08:19:37.903 INFO:teuthology.orchestra.run.trial091.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 3.8 MB/s | 90 kB 00:00 2026-02-25T08:19:37.968 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:56 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:38.033 INFO:teuthology.orchestra.run.trial039.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:38.051 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:38.051 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:38.052 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:38.073 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install librbd1 2026-02-25T08:19:38.335 INFO:teuthology.orchestra.run.trial091.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 52 kB/s | 22 kB 00:00 2026-02-25T08:19:38.393 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:57 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:38.459 INFO:teuthology.orchestra.run.trial039.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:38.476 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:38.477 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:38.477 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:38.498 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install python3-rados 2026-02-25T08:19:38.747 INFO:teuthology.orchestra.run.trial091.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 2.0 MB/s | 862 kB 00:00 2026-02-25T08:19:38.813 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:57 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:38.878 INFO:teuthology.orchestra.run.trial039.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:38.895 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:38.896 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:38.896 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:38.917 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install python3-rgw 2026-02-25T08:19:39.235 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:58 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:39.266 INFO:teuthology.orchestra.run.trial091.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 1.0 MB/s | 551 kB 00:00 2026-02-25T08:19:39.300 INFO:teuthology.orchestra.run.trial039.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:39.317 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:39.318 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:39.318 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:39.328 INFO:teuthology.orchestra.run.trial091.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 316 kB/s | 19 kB 00:00 2026-02-25T08:19:39.340 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install python3-cephfs 2026-02-25T08:19:39.399 INFO:teuthology.orchestra.run.trial091.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 2.1 MB/s | 151 kB 00:00 2026-02-25T08:19:39.502 INFO:teuthology.orchestra.run.trial091.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 5.2 MB/s | 548 kB 00:00 2026-02-25T08:19:39.564 INFO:teuthology.orchestra.run.trial091.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 472 kB/s | 29 kB 00:00 2026-02-25T08:19:39.628 INFO:teuthology.orchestra.run.trial091.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 936 kB/s | 60 kB 00:00 2026-02-25T08:19:39.656 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:58 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:39.692 INFO:teuthology.orchestra.run.trial091.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 686 kB/s | 43 kB 00:00 2026-02-25T08:19:39.721 INFO:teuthology.orchestra.run.trial039.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:39.739 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:39.740 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:39.740 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:39.754 INFO:teuthology.orchestra.run.trial091.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 515 kB/s | 32 kB 00:00 2026-02-25T08:19:39.761 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install python3-rbd 2026-02-25T08:19:39.816 INFO:teuthology.orchestra.run.trial091.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 229 kB/s | 14 kB 00:00 2026-02-25T08:19:39.888 INFO:teuthology.orchestra.run.trial091.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 2.4 MB/s | 173 kB 00:00 2026-02-25T08:19:39.971 INFO:teuthology.orchestra.run.trial091.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2026-02-25T08:19:40.048 INFO:teuthology.orchestra.run.trial091.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 3.2 MB/s | 254 kB 00:00 2026-02-25T08:19:40.081 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:59 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:40.148 INFO:teuthology.orchestra.run.trial039.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:40.166 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:40.167 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:40.167 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:40.188 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install rbd-fuse 2026-02-25T08:19:40.234 INFO:teuthology.orchestra.run.trial091.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2026-02-25T08:19:40.298 INFO:teuthology.orchestra.run.trial091.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 2.2 MB/s | 144 kB 00:00 2026-02-25T08:19:40.361 INFO:teuthology.orchestra.run.trial091.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 2.1 MB/s | 139 kB 00:00 2026-02-25T08:19:40.423 INFO:teuthology.orchestra.run.trial091.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 912 kB/s | 56 kB 00:00 2026-02-25T08:19:40.484 INFO:teuthology.orchestra.run.trial091.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 175 kB/s | 11 kB 00:00 2026-02-25T08:19:40.507 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:10:59 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:40.545 INFO:teuthology.orchestra.run.trial091.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 288 kB/s | 18 kB 00:00 2026-02-25T08:19:40.591 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:40.592 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:40.592 INFO:teuthology.orchestra.run.trial039.stdout: Package Architecture Version Repository Size 2026-02-25T08:19:40.592 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:40.592 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:19:40.592 INFO:teuthology.orchestra.run.trial039.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T08:19:40.592 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:40.592 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:19:40.592 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:40.593 INFO:teuthology.orchestra.run.trial039.stdout:Install 1 Package 2026-02-25T08:19:40.593 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:40.593 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 86 k 2026-02-25T08:19:40.593 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 236 k 2026-02-25T08:19:40.593 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:19:40.606 INFO:teuthology.orchestra.run.trial091.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 378 kB/s | 23 kB 00:00 2026-02-25T08:19:40.668 INFO:teuthology.orchestra.run.trial091.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 320 kB/s | 20 kB 00:00 2026-02-25T08:19:40.730 INFO:teuthology.orchestra.run.trial091.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 317 kB/s | 19 kB 00:00 2026-02-25T08:19:40.791 INFO:teuthology.orchestra.run.trial091.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 431 kB/s | 26 kB 00:00 2026-02-25T08:19:40.873 INFO:teuthology.orchestra.run.trial091.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 12 MB/s | 1.0 MB 00:00 2026-02-25T08:19:40.935 INFO:teuthology.orchestra.run.trial091.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2026-02-25T08:19:40.998 INFO:teuthology.orchestra.run.trial091.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 1.3 MB/s | 86 kB 00:00 2026-02-25T08:19:41.060 INFO:teuthology.orchestra.run.trial091.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 940 kB/s | 58 kB 00:00 2026-02-25T08:19:41.121 INFO:teuthology.orchestra.run.trial091.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 268 kB/s | 16 kB 00:00 2026-02-25T08:19:41.163 INFO:teuthology.orchestra.run.trial039.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 151 kB/s | 86 kB 00:00 2026-02-25T08:19:41.164 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:41.164 INFO:teuthology.orchestra.run.trial039.stdout:Total 151 kB/s | 86 kB 00:00 2026-02-25T08:19:41.164 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:19:41.168 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:19:41.168 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:19:41.183 INFO:teuthology.orchestra.run.trial091.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.4 MB/s | 90 kB 00:00 2026-02-25T08:19:41.199 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:19:41.199 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:19:41.245 INFO:teuthology.orchestra.run.trial091.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 502 kB/s | 31 kB 00:00 2026-02-25T08:19:41.251 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:19:41.269 INFO:teuthology.orchestra.run.trial039.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:41.309 INFO:teuthology.orchestra.run.trial091.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 2.9 MB/s | 188 kB 00:00 2026-02-25T08:19:41.357 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:41.370 INFO:teuthology.orchestra.run.trial091.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 965 kB/s | 59 kB 00:00 2026-02-25T08:19:41.422 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:41.422 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:41.422 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:19:41.422 INFO:teuthology.orchestra.run.trial039.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:41.422 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:41.422 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:41.433 INFO:teuthology.orchestra.run.trial091.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 1.9 MB/s | 125 kB 00:00 2026-02-25T08:19:41.450 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install rbd-mirror 2026-02-25T08:19:41.494 INFO:teuthology.orchestra.run.trial091.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 583 kB/s | 36 kB 00:00 2026-02-25T08:19:41.557 INFO:teuthology.orchestra.run.trial091.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 1.4 MB/s | 86 kB 00:00 2026-02-25T08:19:41.619 INFO:teuthology.orchestra.run.trial091.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 1.4 MB/s | 90 kB 00:00 2026-02-25T08:19:41.682 INFO:teuthology.orchestra.run.trial091.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 778 kB/s | 48 kB 00:00 2026-02-25T08:19:41.743 INFO:teuthology.orchestra.run.trial091.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 361 kB/s | 22 kB 00:00 2026-02-25T08:19:41.773 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:11:00 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:41.804 INFO:teuthology.orchestra.run.trial091.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 326 kB/s | 20 kB 00:00 2026-02-25T08:19:41.856 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:41.856 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:41.857 INFO:teuthology.orchestra.run.trial039.stdout: Package Arch Version Repo Size 2026-02-25T08:19:41.857 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:41.857 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:19:41.857 INFO:teuthology.orchestra.run.trial039.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T08:19:41.857 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:41.857 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:19:41.857 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:41.857 INFO:teuthology.orchestra.run.trial039.stdout:Install 1 Package 2026-02-25T08:19:41.857 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:41.857 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 2.6 M 2026-02-25T08:19:41.858 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 11 M 2026-02-25T08:19:41.858 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:19:53.776 INFO:teuthology.orchestra.run.trial039.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 223 kB/s | 2.6 MB 00:11 2026-02-25T08:19:53.777 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:53.778 INFO:teuthology.orchestra.run.trial039.stdout:Total 223 kB/s | 2.6 MB 00:11 2026-02-25T08:19:53.778 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:19:53.783 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:19:53.783 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:19:53.822 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:19:53.822 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:19:53.985 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:19:53.987 INFO:teuthology.orchestra.run.trial039.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:54.010 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:54.011 INFO:teuthology.orchestra.run.trial039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:19:54.011 INFO:teuthology.orchestra.run.trial039.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T08:19:54.011 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:19:54.011 INFO:teuthology.orchestra.run.trial039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:19:54.011 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:54.367 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:54.368 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:54.368 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:19:54.368 INFO:teuthology.orchestra.run.trial039.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:54.368 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:54.368 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:54.397 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install rbd-nbd 2026-02-25T08:19:54.720 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:11:13 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:54.805 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:54.805 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout: Package Architecture Version Repository Size 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout:Install 1 Package 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 166 k 2026-02-25T08:19:54.806 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 496 k 2026-02-25T08:19:54.807 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:19:55.305 INFO:teuthology.orchestra.run.trial039.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 333 kB/s | 166 kB 00:00 2026-02-25T08:19:55.305 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:19:55.306 INFO:teuthology.orchestra.run.trial039.stdout:Total 333 kB/s | 166 kB 00:00 2026-02-25T08:19:55.306 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:19:55.310 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:19:55.311 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:19:55.342 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:19:55.342 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:19:55.399 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:19:55.416 INFO:teuthology.orchestra.run.trial039.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:55.507 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:55.570 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:19:55.571 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:55.571 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:19:55.571 INFO:teuthology.orchestra.run.trial039.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:19:55.571 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:55.571 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:55.600 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-osd-crimson 2026-02-25T08:19:55.921 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:11:14 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:55.987 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:19:56.005 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:56.006 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:19:56.006 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:19:56.027 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T08:19:56.350 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:11:15 ago on Wed 25 Feb 2026 08:08:41 AM UTC. 2026-02-25T08:19:56.433 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:19:56.434 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:56.434 INFO:teuthology.orchestra.run.trial039.stdout: Package Arch Version Repo Size 2026-02-25T08:19:56.434 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:56.434 INFO:teuthology.orchestra.run.trial039.stdout:Installing: 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout:Installing weak dependencies: 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout:Transaction Summary 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout:================================================================================ 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout:Install 2 Packages 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout:Total download size: 445 M 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout:Installed size: 2.7 G 2026-02-25T08:19:56.435 INFO:teuthology.orchestra.run.trial039.stdout:Downloading Packages: 2026-02-25T08:20:31.347 INFO:teuthology.orchestra.run.trial039.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 353 kB/s | 12 MB 00:34 2026-02-25T08:21:43.900 INFO:teuthology.orchestra.run.trial091.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 240 kB/s | 51 MB 03:36 2026-02-25T08:22:00.548 INFO:teuthology.orchestra.run.trial091.stdout:(96/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 354 kB/s | 88 MB 04:13 2026-02-25T08:22:00.550 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:00.551 INFO:teuthology.orchestra.run.trial091.stdout:Total 902 kB/s | 226 MB 04:16 2026-02-25T08:22:00.921 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:22:00.960 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:22:00.961 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:22:01.766 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:22:01.766 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:22:02.594 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:22:02.596 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T08:22:02.600 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T08:22:02.621 INFO:teuthology.orchestra.run.trial091.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T08:22:02.655 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T08:22:02.661 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T08:22:02.663 INFO:teuthology.orchestra.run.trial091.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T08:22:02.668 INFO:teuthology.orchestra.run.trial091.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T08:22:02.691 INFO:teuthology.orchestra.run.trial091.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T08:22:02.704 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T08:22:02.707 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T08:22:02.713 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T08:22:02.715 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T08:22:02.726 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T08:22:02.732 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T08:22:02.757 INFO:teuthology.orchestra.run.trial091.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T08:22:02.760 INFO:teuthology.orchestra.run.trial091.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T08:22:02.811 INFO:teuthology.orchestra.run.trial091.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T08:22:02.824 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T08:22:02.826 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T08:22:02.832 INFO:teuthology.orchestra.run.trial091.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T08:22:02.837 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T08:22:02.842 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T08:22:02.845 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T08:22:02.860 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T08:22:02.864 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T08:22:02.869 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T08:22:02.880 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T08:22:02.889 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T08:22:02.896 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T08:22:02.915 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T08:22:02.927 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T08:22:02.942 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T08:22:02.947 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T08:22:02.955 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T08:22:02.995 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T08:22:03.324 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T08:22:03.333 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T08:22:03.337 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T08:22:03.343 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T08:22:03.345 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T08:22:03.372 INFO:teuthology.orchestra.run.trial091.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T08:22:03.416 INFO:teuthology.orchestra.run.trial091.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T08:22:03.425 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T08:22:03.430 INFO:teuthology.orchestra.run.trial091.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T08:22:03.434 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T08:22:03.439 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T08:22:03.443 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T08:22:03.450 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T08:22:03.453 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T08:22:03.482 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T08:22:03.491 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T08:22:03.497 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T08:22:03.504 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T08:22:03.539 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T08:22:03.765 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T08:22:03.788 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T08:22:03.792 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T08:22:03.851 INFO:teuthology.orchestra.run.trial091.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T08:22:03.853 INFO:teuthology.orchestra.run.trial091.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T08:22:03.874 INFO:teuthology.orchestra.run.trial091.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T08:22:04.202 INFO:teuthology.orchestra.run.trial091.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T08:22:04.276 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T08:22:04.951 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T08:22:04.968 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T08:22:04.978 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T08:22:04.984 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T08:22:04.998 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T08:22:05.014 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T08:22:05.090 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T08:22:05.098 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T08:22:05.122 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T08:22:05.152 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T08:22:05.203 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T08:22:05.209 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T08:22:05.213 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T08:22:05.214 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T08:22:05.233 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:22:05.233 INFO:teuthology.orchestra.run.trial091.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T08:22:05.233 INFO:teuthology.orchestra.run.trial091.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T08:22:05.233 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:05.242 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:22:05.268 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:22:05.268 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T08:22:05.269 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:05.304 INFO:teuthology.orchestra.run.trial091.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T08:22:05.894 INFO:teuthology.orchestra.run.trial091.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T08:22:11.304 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T08:22:11.367 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T08:22:11.369 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:22:11.383 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:22:11.384 INFO:teuthology.orchestra.run.trial091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:11.384 INFO:teuthology.orchestra.run.trial091.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T08:22:11.384 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:11.394 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:22:11.418 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:22:11.418 INFO:teuthology.orchestra.run.trial091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:11.418 INFO:teuthology.orchestra.run.trial091.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T08:22:11.418 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:22:11.418 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:22:11.418 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:11.474 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:22:11.476 INFO:teuthology.orchestra.run.trial091.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:22:11.485 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T08:22:11.511 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T08:22:11.512 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:22:14.415 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:22:14.422 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:22:14.877 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:22:14.878 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:22:14.889 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:22:14.890 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:22:14.952 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:22:14.992 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T08:22:14.993 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:22:15.014 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:22:15.014 INFO:teuthology.orchestra.run.trial091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:15.014 INFO:teuthology.orchestra.run.trial091.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T08:22:15.014 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:22:15.014 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:22:15.014 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:15.025 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:22:15.239 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:22:15.240 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:22:15.261 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:22:15.261 INFO:teuthology.orchestra.run.trial091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:15.261 INFO:teuthology.orchestra.run.trial091.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T08:22:15.261 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:22:15.262 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:22:15.262 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:15.345 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:22:15.364 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:22:15.364 INFO:teuthology.orchestra.run.trial091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:15.364 INFO:teuthology.orchestra.run.trial091.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T08:22:15.364 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:22:15.365 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:22:15.365 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:15.365 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:22:15.371 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:22:15.396 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:22:16.769 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:22:16.769 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T08:22:16.769 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T08:22:16.769 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T08:22:16.769 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T08:22:16.770 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T08:22:16.771 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T08:22:16.772 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T08:22:16.773 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T08:22:16.774 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T08:22:16.775 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T08:22:16.776 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T08:22:16.777 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T08:22:16.777 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T08:22:16.837 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:16.838 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T08:22:16.839 INFO:teuthology.orchestra.run.trial091.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T08:22:16.840 INFO:teuthology.orchestra.run.trial091.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T08:22:16.841 INFO:teuthology.orchestra.run.trial091.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T08:22:16.842 INFO:teuthology.orchestra.run.trial091.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T08:22:16.843 INFO:teuthology.orchestra.run.trial091.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T08:22:16.844 INFO:teuthology.orchestra.run.trial091.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: zip-3.0-35.el9.x86_64 2026-02-25T08:22:16.845 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:16.846 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:16.910 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-base 2026-02-25T08:22:17.233 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:38 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:17.299 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:17.318 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:17.319 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:17.319 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:17.341 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install cephadm 2026-02-25T08:22:17.655 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:38 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:17.720 INFO:teuthology.orchestra.run.trial091.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:17.739 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:17.740 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:17.740 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:17.760 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:22:18.078 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:39 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:18.162 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:18.163 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:18.163 INFO:teuthology.orchestra.run.trial091.stdout: Package Arch Version Repo Size 2026-02-25T08:22:18.163 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:18.164 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:22:18.164 INFO:teuthology.orchestra.run.trial091.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T08:22:18.164 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:18.164 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:22:18.164 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:18.164 INFO:teuthology.orchestra.run.trial091.stdout:Install 1 Package 2026-02-25T08:22:18.164 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:18.164 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 149 k 2026-02-25T08:22:18.164 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 455 k 2026-02-25T08:22:18.164 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:22:18.811 INFO:teuthology.orchestra.run.trial091.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 231 kB/s | 149 kB 00:00 2026-02-25T08:22:18.811 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:18.811 INFO:teuthology.orchestra.run.trial091.stdout:Total 231 kB/s | 149 kB 00:00 2026-02-25T08:22:18.811 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:22:18.816 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:22:18.816 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:22:18.847 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:22:18.847 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:22:18.903 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:22:18.904 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:22:18.927 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:22:18.927 INFO:teuthology.orchestra.run.trial091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:18.927 INFO:teuthology.orchestra.run.trial091.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T08:22:18.927 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:19.216 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:22:19.216 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:19.216 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:22:19.216 INFO:teuthology.orchestra.run.trial091.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:19.217 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:19.217 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:19.244 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-mgr 2026-02-25T08:22:19.563 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:40 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:19.629 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:19.648 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:19.649 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:19.649 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:19.671 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:22:19.986 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:40 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:20.052 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:20.070 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:20.071 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:20.071 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:20.093 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:22:20.406 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:41 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:20.472 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:20.490 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:20.491 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:20.491 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:20.512 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:22:20.829 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:41 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:20.895 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:20.914 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:20.914 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:20.915 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:20.936 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:22:21.255 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:42 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:21.320 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:21.339 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:21.340 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:21.340 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:21.361 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-osd 2026-02-25T08:22:21.681 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:42 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:21.747 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:21.766 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:21.767 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:21.767 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:21.788 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-osd-classic 2026-02-25T08:22:22.105 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:43 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:22.171 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:22.190 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:22.191 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:22.191 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:22.213 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-fuse 2026-02-25T08:22:22.534 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:43 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:22.618 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:22.619 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:22.619 INFO:teuthology.orchestra.run.trial091.stdout: Package Arch Version Repo Size 2026-02-25T08:22:22.619 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:22.619 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:22:22.619 INFO:teuthology.orchestra.run.trial091.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T08:22:22.620 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:22.620 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:22:22.620 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:22.620 INFO:teuthology.orchestra.run.trial091.stdout:Install 1 Package 2026-02-25T08:22:22.620 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:22.620 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 914 k 2026-02-25T08:22:22.620 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 2.9 M 2026-02-25T08:22:22.620 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:22:26.039 INFO:teuthology.orchestra.run.trial091.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 267 kB/s | 914 kB 00:03 2026-02-25T08:22:26.039 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:26.039 INFO:teuthology.orchestra.run.trial091.stdout:Total 267 kB/s | 914 kB 00:03 2026-02-25T08:22:26.039 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:22:26.044 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:22:26.044 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:22:26.078 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:22:26.078 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:22:26.171 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:22:26.189 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:26.422 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:26.484 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:26.485 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:26.485 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:22:26.485 INFO:teuthology.orchestra.run.trial091.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:26.485 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:26.485 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:26.512 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-volume 2026-02-25T08:22:26.838 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:47 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:26.904 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:22:26.922 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:26.923 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:26.923 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:26.945 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install librados-devel 2026-02-25T08:22:27.263 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:48 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:27.347 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:27.348 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:27.349 INFO:teuthology.orchestra.run.trial091.stdout: Package Arch Version Repo Size 2026-02-25T08:22:27.349 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:27.349 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:22:27.349 INFO:teuthology.orchestra.run.trial091.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T08:22:27.349 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:27.349 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:22:27.349 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:27.349 INFO:teuthology.orchestra.run.trial091.stdout:Install 1 Package 2026-02-25T08:22:27.349 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:27.349 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 124 k 2026-02-25T08:22:27.350 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 434 k 2026-02-25T08:22:27.350 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:22:27.915 INFO:teuthology.orchestra.run.trial091.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 218 kB/s | 124 kB 00:00 2026-02-25T08:22:27.916 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:27.916 INFO:teuthology.orchestra.run.trial091.stdout:Total 218 kB/s | 124 kB 00:00 2026-02-25T08:22:27.916 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:22:27.919 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:22:27.919 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:22:27.950 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:22:27.950 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:22:28.005 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:22:28.023 INFO:teuthology.orchestra.run.trial091.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:28.110 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:28.172 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:28.172 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:28.173 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:22:28.173 INFO:teuthology.orchestra.run.trial091.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:28.173 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:28.173 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:28.201 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install libcephfs2 2026-02-25T08:22:28.521 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:49 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:28.587 INFO:teuthology.orchestra.run.trial091.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:28.606 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:28.607 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:28.607 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:28.629 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install libcephfs-devel 2026-02-25T08:22:28.944 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:49 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout: Package Arch Version Repo Size 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout:Install 1 Package 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 37 k 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 171 k 2026-02-25T08:22:29.030 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:22:29.387 INFO:teuthology.orchestra.run.trial091.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 104 kB/s | 37 kB 00:00 2026-02-25T08:22:29.389 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:29.389 INFO:teuthology.orchestra.run.trial091.stdout:Total 104 kB/s | 37 kB 00:00 2026-02-25T08:22:29.389 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:22:29.391 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:22:29.392 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:22:29.394 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:22:29.394 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:22:29.417 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:22:29.434 INFO:teuthology.orchestra.run.trial091.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:29.493 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:29.555 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:29.555 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:29.555 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:22:29.555 INFO:teuthology.orchestra.run.trial091.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:29.555 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:29.555 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:29.582 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install librados2 2026-02-25T08:22:29.905 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:50 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:29.970 INFO:teuthology.orchestra.run.trial091.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:29.987 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:29.988 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:29.988 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:30.009 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install librbd1 2026-02-25T08:22:30.328 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:51 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:30.394 INFO:teuthology.orchestra.run.trial091.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:30.412 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:30.412 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:30.412 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:30.434 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install python3-rados 2026-02-25T08:22:30.751 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:51 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:30.816 INFO:teuthology.orchestra.run.trial091.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:30.834 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:30.834 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:30.834 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:30.855 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install python3-rgw 2026-02-25T08:22:31.175 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:52 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:31.241 INFO:teuthology.orchestra.run.trial091.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:31.258 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:31.259 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:31.259 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:31.281 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install python3-cephfs 2026-02-25T08:22:31.593 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:52 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:31.658 INFO:teuthology.orchestra.run.trial091.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:31.676 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:31.676 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:31.676 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:31.697 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install python3-rbd 2026-02-25T08:22:32.018 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:53 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:32.084 INFO:teuthology.orchestra.run.trial091.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:32.102 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:32.103 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:32.103 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:32.124 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install rbd-fuse 2026-02-25T08:22:32.444 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:53 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:32.527 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:32.527 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:32.527 INFO:teuthology.orchestra.run.trial091.stdout: Package Architecture Version Repository Size 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout:Install 1 Package 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 86 k 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 236 k 2026-02-25T08:22:32.528 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:22:33.365 INFO:teuthology.orchestra.run.trial091.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 103 kB/s | 86 kB 00:00 2026-02-25T08:22:33.367 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:33.367 INFO:teuthology.orchestra.run.trial091.stdout:Total 103 kB/s | 86 kB 00:00 2026-02-25T08:22:33.368 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:22:33.372 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:22:33.372 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:22:33.403 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:22:33.404 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:22:33.458 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:22:33.477 INFO:teuthology.orchestra.run.trial091.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:33.564 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:33.627 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:33.627 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:33.627 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:22:33.627 INFO:teuthology.orchestra.run.trial091.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:33.628 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:33.628 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:33.656 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install rbd-mirror 2026-02-25T08:22:33.972 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:13:54 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:34.056 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:34.056 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:34.056 INFO:teuthology.orchestra.run.trial091.stdout: Package Arch Version Repo Size 2026-02-25T08:22:34.056 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:34.056 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:22:34.057 INFO:teuthology.orchestra.run.trial091.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T08:22:34.057 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:34.057 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:22:34.057 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:34.057 INFO:teuthology.orchestra.run.trial091.stdout:Install 1 Package 2026-02-25T08:22:34.057 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:34.057 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 2.6 M 2026-02-25T08:22:34.057 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 11 M 2026-02-25T08:22:34.057 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:22:46.137 INFO:teuthology.orchestra.run.trial091.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 220 kB/s | 2.6 MB 00:12 2026-02-25T08:22:46.138 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:46.138 INFO:teuthology.orchestra.run.trial091.stdout:Total 220 kB/s | 2.6 MB 00:12 2026-02-25T08:22:46.138 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:22:46.144 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:22:46.144 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:22:46.182 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:22:46.182 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:22:46.346 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:22:46.347 INFO:teuthology.orchestra.run.trial091.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:46.370 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:46.370 INFO:teuthology.orchestra.run.trial091.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:22:46.370 INFO:teuthology.orchestra.run.trial091.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T08:22:46.370 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:22:46.370 INFO:teuthology.orchestra.run.trial091.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:22:46.370 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:46.705 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:46.705 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:46.705 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:22:46.705 INFO:teuthology.orchestra.run.trial091.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:46.706 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:46.706 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:46.733 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install rbd-nbd 2026-02-25T08:22:47.059 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:14:08 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:47.143 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:47.144 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:47.144 INFO:teuthology.orchestra.run.trial091.stdout: Package Architecture Version Repository Size 2026-02-25T08:22:47.144 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:47.144 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:22:47.144 INFO:teuthology.orchestra.run.trial091.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T08:22:47.145 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:47.145 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:22:47.145 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:47.145 INFO:teuthology.orchestra.run.trial091.stdout:Install 1 Package 2026-02-25T08:22:47.145 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:47.145 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 166 k 2026-02-25T08:22:47.145 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 496 k 2026-02-25T08:22:47.145 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:22:47.749 INFO:teuthology.orchestra.run.trial091.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 275 kB/s | 166 kB 00:00 2026-02-25T08:22:47.749 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:22:47.749 INFO:teuthology.orchestra.run.trial091.stdout:Total 275 kB/s | 166 kB 00:00 2026-02-25T08:22:47.750 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:22:47.754 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:22:47.754 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:22:47.785 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:22:47.786 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:22:47.842 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:22:47.860 INFO:teuthology.orchestra.run.trial091.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:47.947 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:48.010 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:22:48.010 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:48.010 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:22:48.010 INFO:teuthology.orchestra.run.trial091.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:48.010 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:48.010 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:48.039 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-osd-crimson 2026-02-25T08:22:48.361 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:14:09 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:48.427 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:22:48.445 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:48.445 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:22:48.445 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:22:48.467 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T08:22:48.784 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:14:09 ago on Wed 25 Feb 2026 08:08:39 AM UTC. 2026-02-25T08:22:48.869 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:22:48.870 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:48.870 INFO:teuthology.orchestra.run.trial091.stdout: Package Arch Version Repo Size 2026-02-25T08:22:48.870 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:48.870 INFO:teuthology.orchestra.run.trial091.stdout:Installing: 2026-02-25T08:22:48.870 INFO:teuthology.orchestra.run.trial091.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T08:22:48.870 INFO:teuthology.orchestra.run.trial091.stdout:Installing weak dependencies: 2026-02-25T08:22:48.871 INFO:teuthology.orchestra.run.trial091.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T08:22:48.871 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:48.871 INFO:teuthology.orchestra.run.trial091.stdout:Transaction Summary 2026-02-25T08:22:48.871 INFO:teuthology.orchestra.run.trial091.stdout:================================================================================ 2026-02-25T08:22:48.871 INFO:teuthology.orchestra.run.trial091.stdout:Install 2 Packages 2026-02-25T08:22:48.871 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:22:48.871 INFO:teuthology.orchestra.run.trial091.stdout:Total download size: 445 M 2026-02-25T08:22:48.871 INFO:teuthology.orchestra.run.trial091.stdout:Installed size: 2.7 G 2026-02-25T08:22:48.871 INFO:teuthology.orchestra.run.trial091.stdout:Downloading Packages: 2026-02-25T08:23:11.572 INFO:teuthology.orchestra.run.trial091.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 543 kB/s | 12 MB 00:22 2026-02-25T08:34:45.051 INFO:teuthology.orchestra.run.trial039.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 498 kB/s | 433 MB 14:48 2026-02-25T08:34:45.052 INFO:teuthology.orchestra.run.trial039.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:34:45.053 INFO:teuthology.orchestra.run.trial039.stdout:Total 512 kB/s | 445 MB 14:48 2026-02-25T08:34:45.053 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction check 2026-02-25T08:34:45.058 INFO:teuthology.orchestra.run.trial039.stdout:Transaction check succeeded. 2026-02-25T08:34:45.059 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction test 2026-02-25T08:34:46.444 INFO:teuthology.orchestra.run.trial039.stdout:Transaction test succeeded. 2026-02-25T08:34:46.444 INFO:teuthology.orchestra.run.trial039.stdout:Running transaction 2026-02-25T08:34:50.063 INFO:teuthology.orchestra.run.trial039.stdout: Preparing : 1/1 2026-02-25T08:35:12.367 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:35:12.385 INFO:teuthology.orchestra.run.trial039.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:35:13.262 INFO:teuthology.orchestra.run.trial039.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:35:13.262 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:35:13.326 INFO:teuthology.orchestra.run.trial039.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:35:13.326 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:35:13.327 INFO:teuthology.orchestra.run.trial039.stdout:Installed: 2026-02-25T08:35:13.327 INFO:teuthology.orchestra.run.trial039.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:35:13.327 INFO:teuthology.orchestra.run.trial039.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:35:13.327 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:35:13.327 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:35:13.405 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:27.851 INFO:teuthology.orchestra.run.trial091.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 443 kB/s | 433 MB 16:38 2026-02-25T08:39:27.852 INFO:teuthology.orchestra.run.trial091.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:39:27.853 INFO:teuthology.orchestra.run.trial091.stdout:Total 456 kB/s | 445 MB 16:38 2026-02-25T08:39:27.853 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction check 2026-02-25T08:39:27.858 INFO:teuthology.orchestra.run.trial091.stdout:Transaction check succeeded. 2026-02-25T08:39:27.858 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction test 2026-02-25T08:39:29.229 INFO:teuthology.orchestra.run.trial091.stdout:Transaction test succeeded. 2026-02-25T08:39:29.229 INFO:teuthology.orchestra.run.trial091.stdout:Running transaction 2026-02-25T08:39:32.852 INFO:teuthology.orchestra.run.trial091.stdout: Preparing : 1/1 2026-02-25T08:39:55.525 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:39:55.545 INFO:teuthology.orchestra.run.trial091.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:39:57.418 INFO:teuthology.orchestra.run.trial091.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:39:57.418 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:39:57.481 INFO:teuthology.orchestra.run.trial091.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:39:57.481 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:39:57.481 INFO:teuthology.orchestra.run.trial091.stdout:Installed: 2026-02-25T08:39:57.482 INFO:teuthology.orchestra.run.trial091.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:39:57.482 INFO:teuthology.orchestra.run.trial091.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:39:57.482 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:39:57.482 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:39:57.558 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:57.559 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:39:57.724 DEBUG:teuthology.orchestra.run.trial039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:39:57.757 INFO:teuthology.orchestra.run.trial039.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:39:57.757 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:39:57.758 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:39:57.760 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:39:57.915 DEBUG:teuthology.orchestra.run.trial091:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:39:57.936 INFO:teuthology.orchestra.run.trial091.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:39:57.937 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:39:57.937 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:39:57.939 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T08:39:57.940 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:39:57.940 DEBUG:teuthology.orchestra.run.trial039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T08:39:57.968 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:39:57.968 DEBUG:teuthology.orchestra.run.trial091:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T08:39:58.002 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T08:39:58.002 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:39:58.002 DEBUG:teuthology.orchestra.run.trial039:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T08:39:58.033 DEBUG:teuthology.orchestra.run.trial039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T08:39:58.096 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:39:58.096 DEBUG:teuthology.orchestra.run.trial091:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T08:39:58.121 DEBUG:teuthology.orchestra.run.trial091:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T08:39:58.184 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T08:39:58.184 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:39:58.184 DEBUG:teuthology.orchestra.run.trial039:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T08:39:58.209 DEBUG:teuthology.orchestra.run.trial039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T08:39:58.272 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:39:58.272 DEBUG:teuthology.orchestra.run.trial091:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T08:39:58.297 DEBUG:teuthology.orchestra.run.trial091:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T08:39:58.361 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T08:39:58.361 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:39:58.361 DEBUG:teuthology.orchestra.run.trial039:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T08:39:58.385 DEBUG:teuthology.orchestra.run.trial039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T08:39:58.448 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:39:58.448 DEBUG:teuthology.orchestra.run.trial091:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T08:39:58.473 DEBUG:teuthology.orchestra.run.trial091:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T08:39:58.538 INFO:teuthology.run_tasks:Running task pexec... 2026-02-25T08:39:58.550 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-25T08:39:58.550 DEBUG:teuthology.orchestra.run.trial039:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T08:39:58.551 DEBUG:teuthology.orchestra.run.trial091:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T08:39:58.552 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial039.front.sepia.ceph.com 2026-02-25T08:39:58.552 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T08:39:58.580 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial091.front.sepia.ceph.com 2026-02-25T08:39:58.580 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T08:39:58.603 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:58.631 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:58.631 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:39:58.642 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:39:58.642 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:39:58.642 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:39:58.642 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:39:58.645 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:39:58.645 INFO:teuthology.task.install:extra packages: [] 2026-02-25T08:39:58.645 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:39:58.645 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:39:58.646 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:39:58.646 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:39:58.798 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:39:58.798 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:39:58.808 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:39:58.808 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:39:59.140 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:39:59.140 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:39:59.140 DEBUG:teuthology.orchestra.run.trial039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:39:59.165 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T08:39:59.165 DEBUG:teuthology.orchestra.run.trial039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:39:59.184 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:39:59.185 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:39:59.185 DEBUG:teuthology.orchestra.run.trial091:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:39:59.210 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T08:39:59.211 DEBUG:teuthology.orchestra.run.trial091:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:39:59.230 DEBUG:teuthology.orchestra.run.trial039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:39:59.275 DEBUG:teuthology.orchestra.run.trial091:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:39:59.294 DEBUG:teuthology.orchestra.run.trial039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:39:59.339 DEBUG:teuthology.orchestra.run.trial091:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:39:59.349 INFO:teuthology.orchestra.run.trial039.stdout:check_obsoletes = 1 2026-02-25T08:39:59.359 DEBUG:teuthology.orchestra.run.trial039:> sudo yum clean all 2026-02-25T08:39:59.394 INFO:teuthology.orchestra.run.trial091.stdout:check_obsoletes = 1 2026-02-25T08:39:59.404 DEBUG:teuthology.orchestra.run.trial091:> sudo yum clean all 2026-02-25T08:39:59.521 INFO:teuthology.orchestra.run.trial039.stdout:56 files removed 2026-02-25T08:39:59.537 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-radosgw 2026-02-25T08:39:59.564 INFO:teuthology.orchestra.run.trial091.stdout:56 files removed 2026-02-25T08:39:59.579 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-radosgw 2026-02-25T08:40:00.376 INFO:teuthology.orchestra.run.trial039.stdout:ceph packages for x86_64 138 kB/s | 94 kB 00:00 2026-02-25T08:40:00.421 INFO:teuthology.orchestra.run.trial091.stdout:ceph packages for x86_64 137 kB/s | 94 kB 00:00 2026-02-25T08:40:00.981 INFO:teuthology.orchestra.run.trial039.stdout:ceph noarch packages 42 kB/s | 24 kB 00:00 2026-02-25T08:40:01.012 INFO:teuthology.orchestra.run.trial091.stdout:ceph noarch packages 44 kB/s | 24 kB 00:00 2026-02-25T08:40:01.464 INFO:teuthology.orchestra.run.trial039.stdout:ceph source packages 4.5 kB/s | 2.1 kB 00:00 2026-02-25T08:40:01.494 INFO:teuthology.orchestra.run.trial091.stdout:ceph source packages 4.5 kB/s | 2.1 kB 00:00 2026-02-25T08:40:02.242 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-25T08:40:02.396 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.9 MB 00:00 2026-02-25T08:40:05.073 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-25T08:40:05.681 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 27 MB 00:02 2026-02-25T08:40:09.354 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 8.0 MB 00:01 2026-02-25T08:40:11.119 INFO:teuthology.orchestra.run.trial091.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 20 kB 00:01 2026-02-25T08:40:11.420 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 8.0 MB 00:03 2026-02-25T08:40:11.794 INFO:teuthology.orchestra.run.trial091.stdout:Extra Packages for Enterprise Linux 33 MB/s | 20 MB 00:00 2026-02-25T08:40:12.629 INFO:teuthology.orchestra.run.trial039.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 20 kB 00:00 2026-02-25T08:40:14.191 INFO:teuthology.orchestra.run.trial039.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2026-02-25T08:40:15.039 INFO:teuthology.orchestra.run.trial091.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-25T08:40:16.219 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:16.237 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:16.238 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:16.238 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:16.264 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-test 2026-02-25T08:40:16.586 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:01 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:16.653 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:16.671 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:16.672 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:16.672 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:16.694 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph 2026-02-25T08:40:17.014 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:17.079 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:17.096 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:17.097 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:17.097 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:17.118 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-base 2026-02-25T08:40:17.387 INFO:teuthology.orchestra.run.trial039.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-25T08:40:17.438 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:17.504 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:17.522 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:17.523 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:17.523 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:17.544 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install cephadm 2026-02-25T08:40:17.866 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:17.931 INFO:teuthology.orchestra.run.trial091.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:17.949 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:17.950 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:17.950 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:17.972 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:40:18.294 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:18.359 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:18.376 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:18.377 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:18.377 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:18.399 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-mgr 2026-02-25T08:40:18.581 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:18.599 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:18.600 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:18.600 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:18.624 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-test 2026-02-25T08:40:18.716 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:18.783 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:18.800 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:18.801 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:18.801 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:18.822 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:40:18.948 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:01 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:19.014 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:19.032 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:19.033 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:19.033 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:19.054 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph 2026-02-25T08:40:19.143 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:19.211 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:19.229 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:19.230 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:19.230 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:19.251 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:40:19.375 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:19.441 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:19.459 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:19.459 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:19.459 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:19.482 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-base 2026-02-25T08:40:19.567 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:19.633 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:19.651 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:19.651 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:19.651 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:19.672 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:40:19.807 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:19.872 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:19.890 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:19.891 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:19.891 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:19.913 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install cephadm 2026-02-25T08:40:19.992 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:20.058 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:20.076 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:20.077 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:20.077 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:20.098 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:40:20.238 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:20.305 INFO:teuthology.orchestra.run.trial039.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:20.323 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:20.324 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:20.324 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:20.346 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:40:20.417 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:20.484 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:20.502 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:20.502 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:20.502 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:20.524 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-osd 2026-02-25T08:40:20.667 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:20.732 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:20.750 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:20.751 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:20.751 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:20.773 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-mgr 2026-02-25T08:40:20.852 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:20.919 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:20.936 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:20.937 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:20.937 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:20.959 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-osd-classic 2026-02-25T08:40:21.096 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:21.161 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:21.179 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:21.180 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:21.180 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:21.201 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:40:21.282 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:21.348 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:21.366 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:21.366 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:21.366 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:21.389 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-fuse 2026-02-25T08:40:21.524 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:21.590 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:21.607 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:21.608 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:21.608 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:21.630 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:40:21.707 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:21.772 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:21.790 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:21.790 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:21.791 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:21.812 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install ceph-volume 2026-02-25T08:40:21.950 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:22.016 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:22.033 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:22.034 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:22.034 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:22.057 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:40:22.127 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:22.193 INFO:teuthology.orchestra.run.trial091.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:22.211 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:22.212 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:22.212 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:22.233 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install librados-devel 2026-02-25T08:40:22.382 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:22.447 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:22.466 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:22.467 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:22.467 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:22.489 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:40:22.553 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:22.619 INFO:teuthology.orchestra.run.trial091.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:22.637 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:22.638 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:22.638 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:22.660 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install libcephfs2 2026-02-25T08:40:22.810 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:22.878 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:22.896 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:22.896 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:22.897 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:22.918 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-osd 2026-02-25T08:40:22.978 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:23.044 INFO:teuthology.orchestra.run.trial091.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:23.062 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:23.063 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:23.063 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:23.084 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install libcephfs-devel 2026-02-25T08:40:23.242 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:23.307 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:23.325 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:23.325 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:23.325 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:23.347 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-osd-classic 2026-02-25T08:40:23.401 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:23.468 INFO:teuthology.orchestra.run.trial091.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:23.485 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:23.486 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:23.486 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:23.508 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install librados2 2026-02-25T08:40:23.664 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:23.729 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:23.747 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:23.748 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:23.748 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:23.769 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-fuse 2026-02-25T08:40:23.828 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:23.894 INFO:teuthology.orchestra.run.trial091.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:23.912 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:23.913 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:23.913 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:23.935 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install librbd1 2026-02-25T08:40:24.086 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:24.152 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:24.170 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:24.170 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:24.171 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:24.193 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install ceph-volume 2026-02-25T08:40:24.256 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:24.321 INFO:teuthology.orchestra.run.trial091.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:24.339 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:24.339 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:24.339 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:24.361 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install python3-rados 2026-02-25T08:40:24.512 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:24.578 INFO:teuthology.orchestra.run.trial039.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:24.596 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:24.596 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:24.597 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:24.618 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install librados-devel 2026-02-25T08:40:24.680 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:24.746 INFO:teuthology.orchestra.run.trial091.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:24.764 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:24.765 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:24.765 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:24.787 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install python3-rgw 2026-02-25T08:40:24.938 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:25.004 INFO:teuthology.orchestra.run.trial039.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:25.021 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:25.022 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:25.022 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:25.044 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install libcephfs2 2026-02-25T08:40:25.106 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:25.173 INFO:teuthology.orchestra.run.trial091.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:25.190 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:25.191 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:25.191 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:25.213 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install python3-cephfs 2026-02-25T08:40:25.362 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:25.429 INFO:teuthology.orchestra.run.trial039.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:25.447 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:25.447 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:25.447 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:25.470 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install libcephfs-devel 2026-02-25T08:40:25.532 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:25.599 INFO:teuthology.orchestra.run.trial091.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:25.616 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:25.617 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:25.617 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:25.639 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install python3-rbd 2026-02-25T08:40:25.796 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:25.861 INFO:teuthology.orchestra.run.trial039.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:25.879 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:25.879 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:25.880 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:25.902 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install librados2 2026-02-25T08:40:25.961 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:26.026 INFO:teuthology.orchestra.run.trial091.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:26.044 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:26.044 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:26.045 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:26.066 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install rbd-fuse 2026-02-25T08:40:26.223 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:26.290 INFO:teuthology.orchestra.run.trial039.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:26.308 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:26.308 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:26.309 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:26.331 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install librbd1 2026-02-25T08:40:26.383 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:26.448 INFO:teuthology.orchestra.run.trial091.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:26.466 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:26.466 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:26.467 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:26.488 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install rbd-mirror 2026-02-25T08:40:26.649 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:26.715 INFO:teuthology.orchestra.run.trial039.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:26.733 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:26.733 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:26.734 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:26.756 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install python3-rados 2026-02-25T08:40:26.810 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:26.876 INFO:teuthology.orchestra.run.trial091.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:26.894 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:26.895 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:26.895 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:26.917 DEBUG:teuthology.orchestra.run.trial091:> sudo yum -y install rbd-nbd 2026-02-25T08:40:27.076 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:27.141 INFO:teuthology.orchestra.run.trial039.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:27.159 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:27.160 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:27.160 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:27.182 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install python3-rgw 2026-02-25T08:40:27.244 INFO:teuthology.orchestra.run.trial091.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:15 AM UTC. 2026-02-25T08:40:27.310 INFO:teuthology.orchestra.run.trial091.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:27.328 INFO:teuthology.orchestra.run.trial091.stdout:Dependencies resolved. 2026-02-25T08:40:27.328 INFO:teuthology.orchestra.run.trial091.stdout:Nothing to do. 2026-02-25T08:40:27.328 INFO:teuthology.orchestra.run.trial091.stdout:Complete! 2026-02-25T08:40:27.355 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:27.504 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:27.570 INFO:teuthology.orchestra.run.trial039.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:27.588 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:27.588 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:27.589 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:27.610 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install python3-cephfs 2026-02-25T08:40:27.929 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:27.995 INFO:teuthology.orchestra.run.trial039.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:28.013 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:28.014 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:28.014 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:28.035 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install python3-rbd 2026-02-25T08:40:28.359 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:28.425 INFO:teuthology.orchestra.run.trial039.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:28.443 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:28.444 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:28.444 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:28.466 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install rbd-fuse 2026-02-25T08:40:28.786 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:28.851 INFO:teuthology.orchestra.run.trial039.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:28.869 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:28.869 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:28.870 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:28.891 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install rbd-mirror 2026-02-25T08:40:29.213 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:29.279 INFO:teuthology.orchestra.run.trial039.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:29.296 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:29.297 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:29.297 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:29.320 DEBUG:teuthology.orchestra.run.trial039:> sudo yum -y install rbd-nbd 2026-02-25T08:40:29.638 INFO:teuthology.orchestra.run.trial039.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:17 AM UTC. 2026-02-25T08:40:29.703 INFO:teuthology.orchestra.run.trial039.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:29.721 INFO:teuthology.orchestra.run.trial039.stdout:Dependencies resolved. 2026-02-25T08:40:29.722 INFO:teuthology.orchestra.run.trial039.stdout:Nothing to do. 2026-02-25T08:40:29.722 INFO:teuthology.orchestra.run.trial039.stdout:Complete! 2026-02-25T08:40:29.747 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:29.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:29.908 DEBUG:teuthology.orchestra.run.trial039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:29.928 INFO:teuthology.orchestra.run.trial039.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:29.929 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:29.929 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:29.930 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:30.086 DEBUG:teuthology.orchestra.run.trial091:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:30.107 INFO:teuthology.orchestra.run.trial091.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:30.108 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:30.108 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:30.111 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-25T08:40:30.112 INFO:teuthology.run_tasks:Running task ceph... 2026-02-25T08:40:30.216 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-25T08:40:30.216 DEBUG:teuthology.orchestra.run.trial039:> sudo chmod 777 /var/log/ceph 2026-02-25T08:40:30.219 DEBUG:teuthology.orchestra.run.trial091:> sudo chmod 777 /var/log/ceph 2026-02-25T08:40:30.244 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-25T08:40:30.244 DEBUG:teuthology.orchestra.run.trial039:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T08:40:30.283 DEBUG:teuthology.orchestra.run.trial091:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T08:40:30.308 INFO:tasks.ceph:Creating extra log directories... 2026-02-25T08:40:30.308 DEBUG:teuthology.orchestra.run.trial039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T08:40:30.346 DEBUG:teuthology.orchestra.run.trial091:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T08:40:30.392 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-25T08:40:30.392 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-25T08:40:30.392 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062', 'branch': 'wip-logstoremanager', 'description': 'crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} thrashers/{default} thrashosds-health workloads/snaps-few-objects-balanced}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '69062', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}}, 'owner': 'scheduled_matan@soko04.front.sepia.ceph.com', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 869, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d', 'targets': {'trial039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJjhMoFPbq1QANy10SGc4gKqFPs0CVgL+6BZgqTeSLWdZByXSM2OO0fVLvTRc8eeP2WqJ+gF4mmVH28IlFP39H0=', 'trial091.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBItTSov8FyRFZssGD3WHztT8dQGGRGbuj77KeaGVWPcgigCuDtJUoAbThvBCEn4VBYo4zfd63lee9wUFCGANM0s='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'balance_reads': True, 'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-25_07:55:35', 'tube': 'trial', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-25T08:40:30.393 DEBUG:teuthology.orchestra.run.trial039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T08:40:30.410 DEBUG:teuthology.orchestra.run.trial091:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T08:40:30.429 DEBUG:teuthology.orchestra.run.trial039:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T08:40:30.466 DEBUG:teuthology.orchestra.run.trial091:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T08:40:30.493 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:30.493 DEBUG:teuthology.orchestra.run.trial039:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T08:40:30.544 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T08:40:30.544 DEBUG:teuthology.orchestra.run.trial039:> stat /dev/vg_nvme/lv_1 2026-02-25T08:40:30.601 INFO:teuthology.orchestra.run.trial039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T08:40:30.601 INFO:teuthology.orchestra.run.trial039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:30.601 INFO:teuthology.orchestra.run.trial039.stdout:Device: 6h/6d Inode: 1302 Links: 1 2026-02-25T08:40:30.601 INFO:teuthology.orchestra.run.trial039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:30.601 INFO:teuthology.orchestra.run.trial039.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:30.601 INFO:teuthology.orchestra.run.trial039.stdout:Access: 2026-02-25 08:19:54.163541645 +0000 2026-02-25T08:40:30.601 INFO:teuthology.orchestra.run.trial039.stdout:Modify: 2026-02-25 08:09:33.648890529 +0000 2026-02-25T08:40:30.601 INFO:teuthology.orchestra.run.trial039.stdout:Change: 2026-02-25 08:09:33.648890529 +0000 2026-02-25T08:40:30.601 INFO:teuthology.orchestra.run.trial039.stdout: Birth: 2026-02-25 08:09:33.648890529 +0000 2026-02-25T08:40:30.602 DEBUG:teuthology.orchestra.run.trial039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T08:40:30.664 INFO:teuthology.orchestra.run.trial039.stderr:1+0 records in 2026-02-25T08:40:30.664 INFO:teuthology.orchestra.run.trial039.stderr:1+0 records out 2026-02-25T08:40:30.665 INFO:teuthology.orchestra.run.trial039.stderr:512 bytes copied, 0.000155158 s, 3.3 MB/s 2026-02-25T08:40:30.666 DEBUG:teuthology.orchestra.run.trial039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T08:40:30.723 DEBUG:teuthology.orchestra.run.trial039:> stat /dev/vg_nvme/lv_2 2026-02-25T08:40:30.779 INFO:teuthology.orchestra.run.trial039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T08:40:30.780 INFO:teuthology.orchestra.run.trial039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:30.780 INFO:teuthology.orchestra.run.trial039.stdout:Device: 6h/6d Inode: 1303 Links: 1 2026-02-25T08:40:30.780 INFO:teuthology.orchestra.run.trial039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:30.780 INFO:teuthology.orchestra.run.trial039.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:30.780 INFO:teuthology.orchestra.run.trial039.stdout:Access: 2026-02-25 08:19:54.164541624 +0000 2026-02-25T08:40:30.780 INFO:teuthology.orchestra.run.trial039.stdout:Modify: 2026-02-25 08:09:33.648890529 +0000 2026-02-25T08:40:30.780 INFO:teuthology.orchestra.run.trial039.stdout:Change: 2026-02-25 08:09:33.648890529 +0000 2026-02-25T08:40:30.780 INFO:teuthology.orchestra.run.trial039.stdout: Birth: 2026-02-25 08:09:33.648890529 +0000 2026-02-25T08:40:30.781 DEBUG:teuthology.orchestra.run.trial039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T08:40:30.842 INFO:teuthology.orchestra.run.trial039.stderr:1+0 records in 2026-02-25T08:40:30.842 INFO:teuthology.orchestra.run.trial039.stderr:1+0 records out 2026-02-25T08:40:30.842 INFO:teuthology.orchestra.run.trial039.stderr:512 bytes copied, 0.000139078 s, 3.7 MB/s 2026-02-25T08:40:30.843 DEBUG:teuthology.orchestra.run.trial039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T08:40:30.900 DEBUG:teuthology.orchestra.run.trial039:> stat /dev/vg_nvme/lv_3 2026-02-25T08:40:30.956 INFO:teuthology.orchestra.run.trial039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T08:40:30.956 INFO:teuthology.orchestra.run.trial039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:30.956 INFO:teuthology.orchestra.run.trial039.stdout:Device: 6h/6d Inode: 1291 Links: 1 2026-02-25T08:40:30.956 INFO:teuthology.orchestra.run.trial039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:30.956 INFO:teuthology.orchestra.run.trial039.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:30.956 INFO:teuthology.orchestra.run.trial039.stdout:Access: 2026-02-25 08:19:54.164541624 +0000 2026-02-25T08:40:30.956 INFO:teuthology.orchestra.run.trial039.stdout:Modify: 2026-02-25 08:09:33.647890528 +0000 2026-02-25T08:40:30.956 INFO:teuthology.orchestra.run.trial039.stdout:Change: 2026-02-25 08:09:33.647890528 +0000 2026-02-25T08:40:30.956 INFO:teuthology.orchestra.run.trial039.stdout: Birth: 2026-02-25 08:09:33.647890528 +0000 2026-02-25T08:40:30.956 DEBUG:teuthology.orchestra.run.trial039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T08:40:31.018 INFO:teuthology.orchestra.run.trial039.stderr:1+0 records in 2026-02-25T08:40:31.018 INFO:teuthology.orchestra.run.trial039.stderr:1+0 records out 2026-02-25T08:40:31.018 INFO:teuthology.orchestra.run.trial039.stderr:512 bytes copied, 0.000219998 s, 2.3 MB/s 2026-02-25T08:40:31.019 DEBUG:teuthology.orchestra.run.trial039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T08:40:31.076 DEBUG:teuthology.orchestra.run.trial039:> stat /dev/vg_nvme/lv_4 2026-02-25T08:40:31.132 INFO:teuthology.orchestra.run.trial039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T08:40:31.133 INFO:teuthology.orchestra.run.trial039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:31.133 INFO:teuthology.orchestra.run.trial039.stdout:Device: 6h/6d Inode: 1310 Links: 1 2026-02-25T08:40:31.133 INFO:teuthology.orchestra.run.trial039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:31.133 INFO:teuthology.orchestra.run.trial039.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:31.133 INFO:teuthology.orchestra.run.trial039.stdout:Access: 2026-02-25 08:19:54.164541624 +0000 2026-02-25T08:40:31.133 INFO:teuthology.orchestra.run.trial039.stdout:Modify: 2026-02-25 08:09:33.648890529 +0000 2026-02-25T08:40:31.133 INFO:teuthology.orchestra.run.trial039.stdout:Change: 2026-02-25 08:09:33.648890529 +0000 2026-02-25T08:40:31.133 INFO:teuthology.orchestra.run.trial039.stdout: Birth: 2026-02-25 08:09:33.648890529 +0000 2026-02-25T08:40:31.134 DEBUG:teuthology.orchestra.run.trial039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T08:40:31.195 INFO:teuthology.orchestra.run.trial039.stderr:1+0 records in 2026-02-25T08:40:31.195 INFO:teuthology.orchestra.run.trial039.stderr:1+0 records out 2026-02-25T08:40:31.195 INFO:teuthology.orchestra.run.trial039.stderr:512 bytes copied, 0.000209128 s, 2.4 MB/s 2026-02-25T08:40:31.196 DEBUG:teuthology.orchestra.run.trial039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T08:40:31.254 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:31.254 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:40:31.254 DEBUG:teuthology.orchestra.run.trial091:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T08:40:31.273 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T08:40:31.273 DEBUG:teuthology.orchestra.run.trial091:> stat /dev/vg_nvme/lv_1 2026-02-25T08:40:31.329 INFO:teuthology.orchestra.run.trial091.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T08:40:31.329 INFO:teuthology.orchestra.run.trial091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:31.329 INFO:teuthology.orchestra.run.trial091.stdout:Device: 6h/6d Inode: 1300 Links: 1 2026-02-25T08:40:31.329 INFO:teuthology.orchestra.run.trial091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:31.329 INFO:teuthology.orchestra.run.trial091.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:31.330 INFO:teuthology.orchestra.run.trial091.stdout:Access: 2026-02-25 08:22:46.516726769 +0000 2026-02-25T08:40:31.330 INFO:teuthology.orchestra.run.trial091.stdout:Modify: 2026-02-25 08:12:05.010550638 +0000 2026-02-25T08:40:31.330 INFO:teuthology.orchestra.run.trial091.stdout:Change: 2026-02-25 08:12:05.010550638 +0000 2026-02-25T08:40:31.330 INFO:teuthology.orchestra.run.trial091.stdout: Birth: 2026-02-25 08:12:05.010550638 +0000 2026-02-25T08:40:31.330 DEBUG:teuthology.orchestra.run.trial091:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T08:40:31.392 INFO:teuthology.orchestra.run.trial091.stderr:1+0 records in 2026-02-25T08:40:31.393 INFO:teuthology.orchestra.run.trial091.stderr:1+0 records out 2026-02-25T08:40:31.393 INFO:teuthology.orchestra.run.trial091.stderr:512 bytes copied, 0.000157622 s, 3.2 MB/s 2026-02-25T08:40:31.394 DEBUG:teuthology.orchestra.run.trial091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T08:40:31.451 DEBUG:teuthology.orchestra.run.trial091:> stat /dev/vg_nvme/lv_2 2026-02-25T08:40:31.508 INFO:teuthology.orchestra.run.trial091.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T08:40:31.508 INFO:teuthology.orchestra.run.trial091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:31.509 INFO:teuthology.orchestra.run.trial091.stdout:Device: 6h/6d Inode: 1312 Links: 1 2026-02-25T08:40:31.509 INFO:teuthology.orchestra.run.trial091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:31.509 INFO:teuthology.orchestra.run.trial091.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:31.509 INFO:teuthology.orchestra.run.trial091.stdout:Access: 2026-02-25 08:22:46.516726769 +0000 2026-02-25T08:40:31.509 INFO:teuthology.orchestra.run.trial091.stdout:Modify: 2026-02-25 08:12:05.011550635 +0000 2026-02-25T08:40:31.509 INFO:teuthology.orchestra.run.trial091.stdout:Change: 2026-02-25 08:12:05.011550635 +0000 2026-02-25T08:40:31.509 INFO:teuthology.orchestra.run.trial091.stdout: Birth: 2026-02-25 08:12:05.011550635 +0000 2026-02-25T08:40:31.509 DEBUG:teuthology.orchestra.run.trial091:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T08:40:31.572 INFO:teuthology.orchestra.run.trial091.stderr:1+0 records in 2026-02-25T08:40:31.572 INFO:teuthology.orchestra.run.trial091.stderr:1+0 records out 2026-02-25T08:40:31.572 INFO:teuthology.orchestra.run.trial091.stderr:512 bytes copied, 0.000142274 s, 3.6 MB/s 2026-02-25T08:40:31.573 DEBUG:teuthology.orchestra.run.trial091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T08:40:31.631 DEBUG:teuthology.orchestra.run.trial091:> stat /dev/vg_nvme/lv_3 2026-02-25T08:40:31.688 INFO:teuthology.orchestra.run.trial091.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T08:40:31.688 INFO:teuthology.orchestra.run.trial091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:31.688 INFO:teuthology.orchestra.run.trial091.stdout:Device: 6h/6d Inode: 1310 Links: 1 2026-02-25T08:40:31.688 INFO:teuthology.orchestra.run.trial091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:31.688 INFO:teuthology.orchestra.run.trial091.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:31.688 INFO:teuthology.orchestra.run.trial091.stdout:Access: 2026-02-25 08:22:46.516726769 +0000 2026-02-25T08:40:31.688 INFO:teuthology.orchestra.run.trial091.stdout:Modify: 2026-02-25 08:12:05.010550638 +0000 2026-02-25T08:40:31.688 INFO:teuthology.orchestra.run.trial091.stdout:Change: 2026-02-25 08:12:05.010550638 +0000 2026-02-25T08:40:31.688 INFO:teuthology.orchestra.run.trial091.stdout: Birth: 2026-02-25 08:12:05.010550638 +0000 2026-02-25T08:40:31.689 DEBUG:teuthology.orchestra.run.trial091:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T08:40:31.751 INFO:teuthology.orchestra.run.trial091.stderr:1+0 records in 2026-02-25T08:40:31.751 INFO:teuthology.orchestra.run.trial091.stderr:1+0 records out 2026-02-25T08:40:31.751 INFO:teuthology.orchestra.run.trial091.stderr:512 bytes copied, 0.000145029 s, 3.5 MB/s 2026-02-25T08:40:31.753 DEBUG:teuthology.orchestra.run.trial091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T08:40:31.809 DEBUG:teuthology.orchestra.run.trial091:> stat /dev/vg_nvme/lv_4 2026-02-25T08:40:31.866 INFO:teuthology.orchestra.run.trial091.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T08:40:31.866 INFO:teuthology.orchestra.run.trial091.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:31.866 INFO:teuthology.orchestra.run.trial091.stdout:Device: 6h/6d Inode: 1294 Links: 1 2026-02-25T08:40:31.867 INFO:teuthology.orchestra.run.trial091.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:31.867 INFO:teuthology.orchestra.run.trial091.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:31.867 INFO:teuthology.orchestra.run.trial091.stdout:Access: 2026-02-25 08:22:46.516726769 +0000 2026-02-25T08:40:31.867 INFO:teuthology.orchestra.run.trial091.stdout:Modify: 2026-02-25 08:12:05.009550641 +0000 2026-02-25T08:40:31.867 INFO:teuthology.orchestra.run.trial091.stdout:Change: 2026-02-25 08:12:05.009550641 +0000 2026-02-25T08:40:31.867 INFO:teuthology.orchestra.run.trial091.stdout: Birth: 2026-02-25 08:12:05.009550641 +0000 2026-02-25T08:40:31.867 DEBUG:teuthology.orchestra.run.trial091:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T08:40:31.928 INFO:teuthology.orchestra.run.trial091.stderr:1+0 records in 2026-02-25T08:40:31.929 INFO:teuthology.orchestra.run.trial091.stderr:1+0 records out 2026-02-25T08:40:31.929 INFO:teuthology.orchestra.run.trial091.stderr:512 bytes copied, 0.000144238 s, 3.5 MB/s 2026-02-25T08:40:31.930 DEBUG:teuthology.orchestra.run.trial091:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T08:40:31.987 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:31.987 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial091.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-25T08:40:31.987 INFO:tasks.ceph:Generating config... 2026-02-25T08:40:31.990 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-25T08:40:31.990 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-25T08:40:31.990 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-25T08:40:31.990 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-25T08:40:31.990 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-25T08:40:31.991 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] osd delete sleep = 1 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] osd max backfills = 3 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-25T08:40:31.992 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2026-02-25T08:40:31.993 INFO:tasks.ceph:[osd] osd target transaction size = 5 2026-02-25T08:40:31.993 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-25T08:40:31.993 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-25T08:40:31.993 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-25T08:40:31.993 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:31.993 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-25T08:40:31.993 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-25T08:40:31.993 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-25T08:40:31.993 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-25T08:40:31.993 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-25T08:40:31.993 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-25T08:40:31.993 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 2 2026-02-25T08:40:31.994 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 1 2026-02-25T08:40:31.994 INFO:tasks.ceph:[global] osd_pg_log_trim_min = 0 2026-02-25T08:40:31.994 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2026-02-25T08:40:31.994 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2026-02-25T08:40:31.994 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-25T08:40:31.994 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-25T08:40:31.994 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-25T08:40:31.994 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-25T08:40:31.994 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-25T08:40:31.994 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-25T08:40:31.994 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-25T08:40:31.994 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-25T08:40:31.995 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-25T08:40:31.995 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-25T08:40:31.995 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-25T08:40:31.995 INFO:tasks.ceph:Setting up mon.a... 2026-02-25T08:40:31.995 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-25T08:40:32.038 INFO:teuthology.orchestra.run.trial039.stdout:creating /etc/ceph/ceph.keyring 2026-02-25T08:40:32.042 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-25T08:40:32.124 DEBUG:teuthology.orchestra.run.trial039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:32.188 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.39')] 2026-02-25T08:40:32.189 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.39', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-25T08:40:32.190 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:32.190 DEBUG:teuthology.orchestra.run.trial039:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:40:32.245 DEBUG:teuthology.orchestra.run.trial039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.39 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:32.324 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:32.321+0000 7f868445ff40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:32.324 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:32.321+0000 7f868445ff40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:32.324 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:32.321+0000 7f868445ff40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:32.325 INFO:teuthology.orchestra.run.trial039.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:32.326 INFO:teuthology.orchestra.run.trial039.stdout:monmaptool: generated fsid 10023e6d-f681-4f84-90aa-0a6f37c6d62c 2026-02-25T08:40:32.326 INFO:teuthology.orchestra.run.trial039.stdout:setting min_mon_release = tentacle 2026-02-25T08:40:32.326 INFO:teuthology.orchestra.run.trial039.stdout:epoch 0 2026-02-25T08:40:32.326 INFO:teuthology.orchestra.run.trial039.stdout:fsid 10023e6d-f681-4f84-90aa-0a6f37c6d62c 2026-02-25T08:40:32.326 INFO:teuthology.orchestra.run.trial039.stdout:last_changed 2026-02-25T08:40:32.323741+0000 2026-02-25T08:40:32.326 INFO:teuthology.orchestra.run.trial039.stdout:created 2026-02-25T08:40:32.323741+0000 2026-02-25T08:40:32.326 INFO:teuthology.orchestra.run.trial039.stdout:min_mon_release 20 (tentacle) 2026-02-25T08:40:32.326 INFO:teuthology.orchestra.run.trial039.stdout:election_strategy: 1 2026-02-25T08:40:32.326 INFO:teuthology.orchestra.run.trial039.stdout:0: [v2:10.20.193.39:3300/0,v1:10.20.193.39:6789/0] mon.a 2026-02-25T08:40:32.326 INFO:teuthology.orchestra.run.trial039.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-25T08:40:32.327 DEBUG:teuthology.orchestra.run.trial039:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:40:32.382 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 10023e6d-f681-4f84-90aa-0a6f37c6d62c... 2026-02-25T08:40:32.383 DEBUG:teuthology.orchestra.run.trial039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T08:40:32.423 DEBUG:teuthology.orchestra.run.trial091:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T08:40:32.460 INFO:teuthology.orchestra.run.trial039.stdout:[global] 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: chdir = "" 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: auth supported = cephx 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: filestore xattr use omap = true 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: mon clock drift allowed = 1.000 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: osd crush chooseleaf type = 0 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: auth debug = true 2026-02-25T08:40:32.461 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: ms die on old message = true 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: ms die on bug = true 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: mon pg warn max object skew = 0 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: osd pool default size = 2 2026-02-25T08:40:32.462 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: mon osd allow primary affinity = true 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: mon osd allow pg remap = true 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: mon warn on legacy crush tunables = false 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: mon warn on crush straw calc version zero = false 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: mon warn on no sortbitwise = false 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: mon warn on osd down out interval zero = false 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: mon warn on too few osds = false 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: mon_allow_pool_size_one = true 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.463 INFO:teuthology.orchestra.run.trial039.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: osd default data pool replay window = 5 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: mon allow pool delete = true 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: mon cluster log file level = debug 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: debug asserts on shutdown = true 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: mon health detail to clog = false 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: mon host = 10.20.193.39 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: ms client mode = crc 2026-02-25T08:40:32.464 INFO:teuthology.orchestra.run.trial039.stdout: ms cluster mode = crc 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: ms mon client mode = crc 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: ms mon cluster mode = crc 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: ms mon service mode = crc 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: ms service mode = crc 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: osd_max_pg_log_entries = 2 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: osd_min_pg_log_entries = 1 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: osd_pg_log_trim_min = 0 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: osd_pool_default_min_size = 2 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: osd_pool_default_size = 3 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: fsid = 10023e6d-f681-4f84-90aa-0a6f37c6d62c 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout:[osd] 2026-02-25T08:40:32.465 INFO:teuthology.orchestra.run.trial039.stdout: osd journal size = 100 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: osd scrub load threshold = 5.0 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: osd scrub max interval = 120 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: osd mclock profile = high_recovery_ops 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: osd mclock skip benchmark = true 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: osd recover clone overlap = true 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: osd recovery max chunk = 1048576 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: osd debug shutdown = true 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: osd debug op order = true 2026-02-25T08:40:32.466 INFO:teuthology.orchestra.run.trial039.stdout: osd debug verify stray on activate = true 2026-02-25T08:40:32.467 INFO:teuthology.orchestra.run.trial039.stdout: osd debug trim objects = true 2026-02-25T08:40:32.467 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.467 INFO:teuthology.orchestra.run.trial039.stdout: osd open classes on start = true 2026-02-25T08:40:32.467 INFO:teuthology.orchestra.run.trial039.stdout: osd debug pg log writeout = true 2026-02-25T08:40:32.467 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.467 INFO:teuthology.orchestra.run.trial039.stdout: osd deep scrub update digest min age = 30 2026-02-25T08:40:32.467 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.467 INFO:teuthology.orchestra.run.trial039.stdout: osd map max advance = 10 2026-02-25T08:40:32.467 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.467 INFO:teuthology.orchestra.run.trial039.stdout: journal zero on create = true 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: filestore ondisk finisher threads = 3 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: filestore apply finisher threads = 3 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: bdev debug aio = true 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: osd debug misdirected ops = true 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: debug monc = 20 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: crimson cpu num = 4 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: crimson osd obc lru size = 10 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: debug ms = 20 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: debug osd = 20 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore = 20 2026-02-25T08:40:32.468 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore backref = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore cache = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore cleaner = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore device = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore epm = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore fixedkv tree = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore journal = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore lba = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore odata = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore omap = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore onode = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore t = 20 2026-02-25T08:40:32.469 INFO:teuthology.orchestra.run.trial039.stdout: debug seastore tm = 20 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: osd blocked scrub grace period = 3600 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: osd debug reject backfill probability = 0.3 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: osd delete sleep = 1 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: osd max backfills = 3 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: osd max markdown count = 1000 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: osd objectstore = seastore 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: osd scrub min interval = 60 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: osd shutdown pgref assert = True 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: osd snap trim sleep = 2 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: osd target transaction size = 5 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: seastore cachepin size pershard = 64M 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: seastore cachepin type = 2Q 2026-02-25T08:40:32.470 INFO:teuthology.orchestra.run.trial039.stdout: seastore max concurrent transactions = 8 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout:[mgr] 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout: debug ms = 1 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout: debug mgr = 20 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout: debug mon = 20 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout: debug auth = 20 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout: mgr/telemetry/nag = false 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout:[mon] 2026-02-25T08:40:32.471 INFO:teuthology.orchestra.run.trial039.stdout: debug ms = 1 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: debug mon = 20 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: debug paxos = 20 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: debug auth = 20 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: mon data avail warn = 5 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: mon mgr mkfs grace = 240 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: mon osd reporter subtree level = osd 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: mon osd prime pg temp = true 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T08:40:32.472 INFO:teuthology.orchestra.run.trial039.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: # 1m isn't quite enough 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: mon_down_mkfs_grace = 2m 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: mon_warn_on_filestore_osds = false 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: mon min osdmap epochs = 50 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: mon osdmap full prune interval = 2 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: mon osdmap full prune min = 15 2026-02-25T08:40:32.473 INFO:teuthology.orchestra.run.trial039.stdout: mon osdmap full prune txsize = 2 2026-02-25T08:40:32.474 INFO:teuthology.orchestra.run.trial039.stdout: osd pool default crimson = True 2026-02-25T08:40:32.474 INFO:teuthology.orchestra.run.trial039.stdout: paxos service trim min = 10 2026-02-25T08:40:32.474 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:40:32.474 INFO:teuthology.orchestra.run.trial039.stdout:[client] 2026-02-25T08:40:32.474 INFO:teuthology.orchestra.run.trial039.stdout: rgw cache enabled = true 2026-02-25T08:40:32.474 INFO:teuthology.orchestra.run.trial039.stdout: rgw enable ops log = true 2026-02-25T08:40:32.474 INFO:teuthology.orchestra.run.trial039.stdout: rgw enable usage log = true 2026-02-25T08:40:32.474 INFO:teuthology.orchestra.run.trial039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T08:40:32.474 INFO:teuthology.orchestra.run.trial039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T08:40:32.474 INFO:teuthology.orchestra.run.trial039.stdout:[mon.a] 2026-02-25T08:40:32.476 INFO:teuthology.orchestra.run.trial091.stdout:[global] 2026-02-25T08:40:32.476 INFO:teuthology.orchestra.run.trial091.stdout: chdir = "" 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: auth supported = cephx 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: filestore xattr use omap = true 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: mon clock drift allowed = 1.000 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: osd crush chooseleaf type = 0 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: auth debug = true 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: ms die on old message = true 2026-02-25T08:40:32.477 INFO:teuthology.orchestra.run.trial091.stdout: ms die on bug = true 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: mon pg warn max object skew = 0 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: osd pool default size = 2 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: mon osd allow primary affinity = true 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: mon osd allow pg remap = true 2026-02-25T08:40:32.478 INFO:teuthology.orchestra.run.trial091.stdout: mon warn on legacy crush tunables = false 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: mon warn on crush straw calc version zero = false 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: mon warn on no sortbitwise = false 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: mon warn on osd down out interval zero = false 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: mon warn on too few osds = false 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: mon_allow_pool_size_one = true 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: osd default data pool replay window = 5 2026-02-25T08:40:32.479 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: mon allow pool delete = true 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: mon cluster log file level = debug 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: debug asserts on shutdown = true 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: mon health detail to clog = false 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: mon host = 10.20.193.39 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: ms client mode = crc 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: ms cluster mode = crc 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: ms mon client mode = crc 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: ms mon cluster mode = crc 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: ms mon service mode = crc 2026-02-25T08:40:32.480 INFO:teuthology.orchestra.run.trial091.stdout: ms service mode = crc 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: osd_max_pg_log_entries = 2 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: osd_min_pg_log_entries = 1 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: osd_pg_log_trim_min = 0 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: osd_pool_default_min_size = 2 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: osd_pool_default_size = 3 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: fsid = 10023e6d-f681-4f84-90aa-0a6f37c6d62c 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout:[osd] 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: osd journal size = 100 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: osd scrub load threshold = 5.0 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: osd scrub max interval = 120 2026-02-25T08:40:32.481 INFO:teuthology.orchestra.run.trial091.stdout: osd mclock profile = high_recovery_ops 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: osd mclock skip benchmark = true 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: osd recover clone overlap = true 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: osd recovery max chunk = 1048576 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: osd debug shutdown = true 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: osd debug op order = true 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: osd debug verify stray on activate = true 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: osd debug trim objects = true 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: osd open classes on start = true 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: osd debug pg log writeout = true 2026-02-25T08:40:32.482 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: osd deep scrub update digest min age = 30 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: osd map max advance = 10 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: journal zero on create = true 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: filestore ondisk finisher threads = 3 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: filestore apply finisher threads = 3 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: bdev debug aio = true 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: osd debug misdirected ops = true 2026-02-25T08:40:32.483 INFO:teuthology.orchestra.run.trial091.stdout: debug monc = 20 2026-02-25T08:40:32.484 INFO:teuthology.orchestra.run.trial091.stdout: crimson cpu num = 4 2026-02-25T08:40:32.484 INFO:teuthology.orchestra.run.trial091.stdout: crimson osd obc lru size = 10 2026-02-25T08:40:32.484 INFO:teuthology.orchestra.run.trial091.stdout: debug ms = 20 2026-02-25T08:40:32.484 INFO:teuthology.orchestra.run.trial091.stdout: debug osd = 20 2026-02-25T08:40:32.484 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore = 20 2026-02-25T08:40:32.484 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore backref = 20 2026-02-25T08:40:32.484 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore cache = 20 2026-02-25T08:40:32.484 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore cleaner = 20 2026-02-25T08:40:32.484 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore device = 20 2026-02-25T08:40:32.484 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore epm = 20 2026-02-25T08:40:32.485 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore fixedkv tree = 20 2026-02-25T08:40:32.485 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore journal = 20 2026-02-25T08:40:32.485 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore lba = 20 2026-02-25T08:40:32.485 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore odata = 20 2026-02-25T08:40:32.485 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore omap = 20 2026-02-25T08:40:32.485 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore onode = 20 2026-02-25T08:40:32.485 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore t = 20 2026-02-25T08:40:32.485 INFO:teuthology.orchestra.run.trial091.stdout: debug seastore tm = 20 2026-02-25T08:40:32.485 INFO:teuthology.orchestra.run.trial091.stdout: osd blocked scrub grace period = 3600 2026-02-25T08:40:32.485 INFO:teuthology.orchestra.run.trial091.stdout: osd debug reject backfill probability = 0.3 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: osd delete sleep = 1 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: osd max backfills = 3 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: osd max markdown count = 1000 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: osd objectstore = seastore 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: osd scrub min interval = 60 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: osd shutdown pgref assert = True 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: osd snap trim sleep = 2 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: osd target transaction size = 5 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: seastore cachepin size pershard = 64M 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: seastore cachepin type = 2Q 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: seastore max concurrent transactions = 8 2026-02-25T08:40:32.486 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout:[mgr] 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: debug ms = 1 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: debug mgr = 20 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: debug mon = 20 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: debug auth = 20 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: mgr/telemetry/nag = false 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout:[mon] 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: debug ms = 1 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: debug mon = 20 2026-02-25T08:40:32.487 INFO:teuthology.orchestra.run.trial091.stdout: debug paxos = 20 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: debug auth = 20 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: mon data avail warn = 5 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: mon mgr mkfs grace = 240 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: mon osd reporter subtree level = osd 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: mon osd prime pg temp = true 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T08:40:32.488 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: # 1m isn't quite enough 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: mon_down_mkfs_grace = 2m 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: mon_warn_on_filestore_osds = false 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: mon min osdmap epochs = 50 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: mon osdmap full prune interval = 2 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: mon osdmap full prune min = 15 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: mon osdmap full prune txsize = 2 2026-02-25T08:40:32.489 INFO:teuthology.orchestra.run.trial091.stdout: osd pool default crimson = True 2026-02-25T08:40:32.490 INFO:teuthology.orchestra.run.trial091.stdout: paxos service trim min = 10 2026-02-25T08:40:32.490 INFO:teuthology.orchestra.run.trial091.stdout: 2026-02-25T08:40:32.490 INFO:teuthology.orchestra.run.trial091.stdout:[client] 2026-02-25T08:40:32.490 INFO:teuthology.orchestra.run.trial091.stdout: rgw cache enabled = true 2026-02-25T08:40:32.490 INFO:teuthology.orchestra.run.trial091.stdout: rgw enable ops log = true 2026-02-25T08:40:32.490 INFO:teuthology.orchestra.run.trial091.stdout: rgw enable usage log = true 2026-02-25T08:40:32.490 INFO:teuthology.orchestra.run.trial091.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T08:40:32.490 INFO:teuthology.orchestra.run.trial091.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T08:40:32.490 INFO:teuthology.orchestra.run.trial091.stdout:[mon.a] 2026-02-25T08:40:32.491 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-25T08:40:32.491 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-25T08:40:32.566 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-25T08:40:32.566 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:32.566 DEBUG:teuthology.orchestra.run.trial039:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-25T08:40:32.583 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:32.583 DEBUG:teuthology.orchestra.run.trial039:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-25T08:40:32.638 INFO:tasks.ceph:Sending monmap to node ubuntu@trial039.front.sepia.ceph.com 2026-02-25T08:40:32.639 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:32.639 DEBUG:teuthology.orchestra.run.trial039:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T08:40:32.639 DEBUG:teuthology.orchestra.run.trial039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:32.712 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:32.712 DEBUG:teuthology.orchestra.run.trial039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:32.768 INFO:tasks.ceph:Sending monmap to node ubuntu@trial091.front.sepia.ceph.com 2026-02-25T08:40:32.769 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:40:32.769 DEBUG:teuthology.orchestra.run.trial091:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T08:40:32.769 DEBUG:teuthology.orchestra.run.trial091:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:32.804 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:40:32.804 DEBUG:teuthology.orchestra.run.trial091:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:32.860 INFO:tasks.ceph:Setting up mon nodes... 2026-02-25T08:40:32.860 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-25T08:40:32.860 DEBUG:teuthology.orchestra.run.trial091:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:40:32.951 INFO:teuthology.orchestra.run.trial091.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:40:32.953 INFO:tasks.ceph:Setting up mds nodes... 2026-02-25T08:40:32.954 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-25T08:40:32.954 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-25T08:40:32.997 INFO:teuthology.orchestra.run.trial039.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-25T08:40:33.009 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2026-02-25T08:40:33.052 INFO:teuthology.orchestra.run.trial091.stdout:creating /etc/ceph/ceph.client.1.keyring 2026-02-25T08:40:33.063 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-25T08:40:33.064 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial091.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-25T08:40:33.064 DEBUG:teuthology.orchestra.run.trial039:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:33.089 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:33.089 INFO:tasks.ceph:role: osd.0 2026-02-25T08:40:33.089 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial039.front.sepia.ceph.com 2026-02-25T08:40:33.089 DEBUG:teuthology.orchestra.run.trial039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T08:40:33.154 INFO:teuthology.orchestra.run.trial039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:33.155 INFO:teuthology.orchestra.run.trial039.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:33.155 INFO:teuthology.orchestra.run.trial039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:33.155 INFO:teuthology.orchestra.run.trial039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:33.155 INFO:teuthology.orchestra.run.trial039.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:33.155 INFO:teuthology.orchestra.run.trial039.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:33.155 INFO:teuthology.orchestra.run.trial039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:33.155 INFO:teuthology.orchestra.run.trial039.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:33.155 INFO:teuthology.orchestra.run.trial039.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:33.155 INFO:teuthology.orchestra.run.trial039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:33.726 INFO:teuthology.orchestra.run.trial039.stdout:Discarding blocks...Done. 2026-02-25T08:40:33.728 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial039.front.sepia.ceph.com -o noatime 2026-02-25T08:40:33.729 DEBUG:teuthology.orchestra.run.trial039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:33.797 DEBUG:teuthology.orchestra.run.trial039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:33.865 DEBUG:teuthology.orchestra.run.trial039:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:33.929 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:33.929 INFO:tasks.ceph:role: osd.1 2026-02-25T08:40:33.929 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial039.front.sepia.ceph.com 2026-02-25T08:40:33.929 DEBUG:teuthology.orchestra.run.trial039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T08:40:33.994 INFO:teuthology.orchestra.run.trial039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:33.995 INFO:teuthology.orchestra.run.trial039.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:33.995 INFO:teuthology.orchestra.run.trial039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:33.995 INFO:teuthology.orchestra.run.trial039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:33.995 INFO:teuthology.orchestra.run.trial039.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:33.995 INFO:teuthology.orchestra.run.trial039.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:33.995 INFO:teuthology.orchestra.run.trial039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:33.995 INFO:teuthology.orchestra.run.trial039.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:33.995 INFO:teuthology.orchestra.run.trial039.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:33.995 INFO:teuthology.orchestra.run.trial039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:34.563 INFO:teuthology.orchestra.run.trial039.stdout:Discarding blocks...Done. 2026-02-25T08:40:34.565 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial039.front.sepia.ceph.com -o noatime 2026-02-25T08:40:34.565 DEBUG:teuthology.orchestra.run.trial039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:34.633 DEBUG:teuthology.orchestra.run.trial039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:34.699 DEBUG:teuthology.orchestra.run.trial039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:34.850 INFO:teuthology.orchestra.run.trial039.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:34.906 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:34,905 seastar - Reactor backend: linux-aio 2026-02-25T08:40:34.923 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:34,922 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:34.923 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:34,922 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:34.925 INFO:teuthology.orchestra.run.trial039.stderr:==39626==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:34.982 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:34,981 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:34.982 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:34,981 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:34.982 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:34,981 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:34.982 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:34,981 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:40:34.982 INFO:teuthology.orchestra.run.trial039.stderr:DEBUG 2026-02-25 08:40:34,981 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:37.055 INFO:teuthology.orchestra.run.trial039.stderr:----------------------------------------------------- 2026-02-25T08:40:37.055 INFO:teuthology.orchestra.run.trial039.stderr:Suppressions used: 2026-02-25T08:40:37.055 INFO:teuthology.orchestra.run.trial039.stderr: count bytes template 2026-02-25T08:40:37.055 INFO:teuthology.orchestra.run.trial039.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:37.056 INFO:teuthology.orchestra.run.trial039.stderr:----------------------------------------------------- 2026-02-25T08:40:37.056 INFO:teuthology.orchestra.run.trial039.stderr: 2026-02-25T08:40:37.075 INFO:teuthology.orchestra.run.trial039.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:37.075 INFO:teuthology.orchestra.run.trial039.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:37.117 INFO:teuthology.orchestra.run.trial039.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:37.117 INFO:teuthology.orchestra.run.trial039.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:37.124 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:37,123 seastar - Reactor backend: linux-aio 2026-02-25T08:40:37.134 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:37,133 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:37.134 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:37,133 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:37.138 INFO:teuthology.orchestra.run.trial039.stderr:==39624==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:37.138 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:37,137 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:37.197 INFO:teuthology.orchestra.run.trial039.stderr:DEBUG 2026-02-25 08:40:37,197 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:37.281 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xcc280b 0x10a2700 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.281 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9dd7fea 0x9eec193 0x9f18680 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.281 INFO:teuthology.orchestra.run.trial039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff94dd4a03 0xffffffff94dd500b 0xffffffff94a910d2 0xffffffff95750451 0xffffffff95800ba2 2026-02-25T08:40:37.282 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcaef /lib64/libasan.so.8+0xedbe8 0xe40f5d 0x9ed9f33 0x9eeae50 0x9eeb09e 0x9eeb14a 0x9f13269 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.313 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0xb8672c 0xc7e332 0x9610617 0x96109f1 0x9610b50 0xa7f3802 0xa7fcfb6 0x95d7664 0x962a4e7 0x90f061c 0x90f0af2 0x916ab89 0x7a59cdd 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:37.357 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x35fc1ca 0x35fc215 0x3618092 0x9e33da7 0x9e33f3f 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1568c 0x9f18413 0x9ed89e1 0x9ed79e1 0x7f64059 0x7fb93c0 0x9121733 0x91240a1 0x7857565 0x7a67c92 0x704540f 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.380 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129764 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:37.421 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521a 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.422 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.422 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.547 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9e21d33 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.547 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d5 /lib64/libasan.so.8+0x157c7 0x10a2823 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.547 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9f09170 0x9f097c6 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.613 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.629 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a1526e 0x2a69674 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:37.645 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151b9 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:37.645 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a695a0 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:39.819 INFO:teuthology.orchestra.run.trial039.stderr:----------------------------------------------------- 2026-02-25T08:40:39.819 INFO:teuthology.orchestra.run.trial039.stderr:Suppressions used: 2026-02-25T08:40:39.819 INFO:teuthology.orchestra.run.trial039.stderr: count bytes template 2026-02-25T08:40:39.819 INFO:teuthology.orchestra.run.trial039.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:39.819 INFO:teuthology.orchestra.run.trial039.stderr:----------------------------------------------------- 2026-02-25T08:40:39.819 INFO:teuthology.orchestra.run.trial039.stderr: 2026-02-25T08:40:39.857 DEBUG:teuthology.orchestra.run.trial039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:39.882 DEBUG:teuthology.orchestra.run.trial039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:40.034 INFO:teuthology.orchestra.run.trial039.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:40.091 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:40,091 seastar - Reactor backend: linux-aio 2026-02-25T08:40:40.101 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:40,100 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:40.101 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:40,100 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:40.103 INFO:teuthology.orchestra.run.trial039.stderr:==39692==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:40.160 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:40,159 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:40.160 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:40,159 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:40.160 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:40,159 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:40.160 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:40,159 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:40:40.160 INFO:teuthology.orchestra.run.trial039.stderr:DEBUG 2026-02-25 08:40:40,159 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:42.232 INFO:teuthology.orchestra.run.trial039.stderr:----------------------------------------------------- 2026-02-25T08:40:42.232 INFO:teuthology.orchestra.run.trial039.stderr:Suppressions used: 2026-02-25T08:40:42.232 INFO:teuthology.orchestra.run.trial039.stderr: count bytes template 2026-02-25T08:40:42.232 INFO:teuthology.orchestra.run.trial039.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:42.232 INFO:teuthology.orchestra.run.trial039.stderr:----------------------------------------------------- 2026-02-25T08:40:42.232 INFO:teuthology.orchestra.run.trial039.stderr: 2026-02-25T08:40:42.252 INFO:teuthology.orchestra.run.trial039.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:42.252 INFO:teuthology.orchestra.run.trial039.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:42.294 INFO:teuthology.orchestra.run.trial039.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:42.294 INFO:teuthology.orchestra.run.trial039.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:42.301 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:42,300 seastar - Reactor backend: linux-aio 2026-02-25T08:40:42.316 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:42,315 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:42.316 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:42,315 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:42.320 INFO:teuthology.orchestra.run.trial039.stderr:==39690==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:42.320 INFO:teuthology.orchestra.run.trial039.stderr:INFO 2026-02-25 08:40:42,319 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:42.382 INFO:teuthology.orchestra.run.trial039.stderr:DEBUG 2026-02-25 08:40:42,381 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:42.462 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xb7ac87 0x10a27ac 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.462 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x10a260f 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.462 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x113812a 0xf4792d 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9f02d0e 0x9f03487 0x9f03972 0x9f03d05 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.495 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf4e8f /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x3171063 0x31746ab 0x33843c3 0x338465f 0x33be345 0x33bef92 0x340ac01 0x340c709 0x340d50e 0xe77777 0xe77a30 0xe77bbf 0x90f0536 0x90f0af2 0x916ab89 0x7a59cdd 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:42.538 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1006a8 /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x10a2823 0xf47938 0x9e26106 0x9e26a45 0x9f12a15 0x9f15a0e 0x9f18413 0x9ed89e1 0x9ed79e1 0x7f64059 0x7fb93c0 0x9121733 0x91240a1 0x7857565 0x7a67c92 0x704540f 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.538 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x35ed461 0x35fc124 0x35fc215 0x3618092 0x9e33da7 0x9e33eaf 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1568c 0x9f18413 0x9ed89e1 0x9ed79e1 0x7f64059 0x7fb93c0 0x9121733 0x91240a1 0x7857565 0x7a67c92 0x704540f 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.561 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a6955f 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:42.603 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a68f88 0x7d56d00 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.603 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521a 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.603 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6585 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.729 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x31826be 0x9e21e0a 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.730 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6583 /lib64/libasan.so.8+0x15197 0x318273f 0x9e21e0a 0x9e22e1c 0x9e2338a 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.730 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x31812d9 0x9e21b19 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.796 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a15256 0x2a69674 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.812 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:42.827 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a689c9 0x7d56cf3 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:42.828 INFO:teuthology.orchestra.run.trial039.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151f5 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:44.993 INFO:teuthology.orchestra.run.trial039.stderr:----------------------------------------------------- 2026-02-25T08:40:44.993 INFO:teuthology.orchestra.run.trial039.stderr:Suppressions used: 2026-02-25T08:40:44.994 INFO:teuthology.orchestra.run.trial039.stderr: count bytes template 2026-02-25T08:40:44.994 INFO:teuthology.orchestra.run.trial039.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:44.994 INFO:teuthology.orchestra.run.trial039.stderr:----------------------------------------------------- 2026-02-25T08:40:44.994 INFO:teuthology.orchestra.run.trial039.stderr: 2026-02-25T08:40:45.030 DEBUG:teuthology.orchestra.run.trial039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:45.055 DEBUG:teuthology.orchestra.run.trial091:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:45.083 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:45.083 INFO:tasks.ceph:role: osd.2 2026-02-25T08:40:45.083 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial091.front.sepia.ceph.com 2026-02-25T08:40:45.083 DEBUG:teuthology.orchestra.run.trial091:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T08:40:45.149 INFO:teuthology.orchestra.run.trial091.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:45.149 INFO:teuthology.orchestra.run.trial091.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:45.149 INFO:teuthology.orchestra.run.trial091.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:45.149 INFO:teuthology.orchestra.run.trial091.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:45.149 INFO:teuthology.orchestra.run.trial091.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:45.149 INFO:teuthology.orchestra.run.trial091.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:45.150 INFO:teuthology.orchestra.run.trial091.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:45.150 INFO:teuthology.orchestra.run.trial091.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:45.150 INFO:teuthology.orchestra.run.trial091.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:45.150 INFO:teuthology.orchestra.run.trial091.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:45.724 INFO:teuthology.orchestra.run.trial091.stdout:Discarding blocks...Done. 2026-02-25T08:40:45.727 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial091.front.sepia.ceph.com -o noatime 2026-02-25T08:40:45.727 DEBUG:teuthology.orchestra.run.trial091:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:45.795 DEBUG:teuthology.orchestra.run.trial091:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:45.863 DEBUG:teuthology.orchestra.run.trial091:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-25T08:40:45.927 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:45.928 INFO:tasks.ceph:role: osd.3 2026-02-25T08:40:45.928 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial091.front.sepia.ceph.com 2026-02-25T08:40:45.928 DEBUG:teuthology.orchestra.run.trial091:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T08:40:45.993 INFO:teuthology.orchestra.run.trial091.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:45.994 INFO:teuthology.orchestra.run.trial091.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:45.994 INFO:teuthology.orchestra.run.trial091.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:45.994 INFO:teuthology.orchestra.run.trial091.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:45.994 INFO:teuthology.orchestra.run.trial091.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:45.994 INFO:teuthology.orchestra.run.trial091.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:45.994 INFO:teuthology.orchestra.run.trial091.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:45.994 INFO:teuthology.orchestra.run.trial091.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:45.994 INFO:teuthology.orchestra.run.trial091.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:45.994 INFO:teuthology.orchestra.run.trial091.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:46.553 INFO:teuthology.orchestra.run.trial091.stdout:Discarding blocks...Done. 2026-02-25T08:40:46.555 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial091.front.sepia.ceph.com -o noatime 2026-02-25T08:40:46.556 DEBUG:teuthology.orchestra.run.trial091:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2026-02-25T08:40:46.624 DEBUG:teuthology.orchestra.run.trial091:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-25T08:40:46.690 DEBUG:teuthology.orchestra.run.trial091:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:46.840 INFO:teuthology.orchestra.run.trial091.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:46.897 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:46,897 seastar - Reactor backend: linux-aio 2026-02-25T08:40:46.910 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:46,910 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:46.911 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:46,910 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:46.912 INFO:teuthology.orchestra.run.trial091.stderr:==39079==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:46.969 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:46,968 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:46.969 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:46,968 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:46.969 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:46,968 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:46.969 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:46,968 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:40:46.969 INFO:teuthology.orchestra.run.trial091.stderr:DEBUG 2026-02-25 08:40:46,969 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:48.981 INFO:teuthology.orchestra.run.trial091.stderr:----------------------------------------------------- 2026-02-25T08:40:48.981 INFO:teuthology.orchestra.run.trial091.stderr:Suppressions used: 2026-02-25T08:40:48.981 INFO:teuthology.orchestra.run.trial091.stderr: count bytes template 2026-02-25T08:40:48.982 INFO:teuthology.orchestra.run.trial091.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:48.982 INFO:teuthology.orchestra.run.trial091.stderr:----------------------------------------------------- 2026-02-25T08:40:48.982 INFO:teuthology.orchestra.run.trial091.stderr: 2026-02-25T08:40:49.000 INFO:teuthology.orchestra.run.trial091.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:49.000 INFO:teuthology.orchestra.run.trial091.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:49.045 INFO:teuthology.orchestra.run.trial091.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:49.045 INFO:teuthology.orchestra.run.trial091.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:49.052 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:49,052 seastar - Reactor backend: linux-aio 2026-02-25T08:40:49.063 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:49,063 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:49.063 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:49,063 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:49.067 INFO:teuthology.orchestra.run.trial091.stderr:==39077==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:49.067 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:49,067 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:49.126 INFO:teuthology.orchestra.run.trial091.stderr:DEBUG 2026-02-25 08:40:49,126 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:49.212 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15835 0x35fc095 0x35fc215 0x3618092 0x9e33da7 0x9e33ed4 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.212 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc250 /lib64/libasan.so.8+0xc553 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x755a777 0x7395f10 0x73e6c79 0x731e88c 0x78b693f 0x7ba17e8 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.212 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521e 0x11380c8 0xf4792d 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.244 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb04bf /lib64/libasan.so.8+0xb1437 0xb7ac87 0x10a27ac 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7b90d7c 0x7ba5441 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:49.286 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cbef /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x35fc095 0x35fc215 0x3618092 0x9e33da7 0x9e33eaf 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1568c 0x9f18413 0x9ed89e1 0x9ed79e1 0x7f64059 0x7fb93c0 0x9121733 0x91240a1 0x7857565 0x7a67c92 0x704540f 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.287 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 66 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191ccd /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x1351ac0 0x2a6ecc8 0x7f68eec 0x7f694a2 0x7f695ce 0x7f6c143 0x7f6cd97 0x7f6d058 0x7e5013f 0x7fb8d74 0x9121733 0x91240a1 0x7857565 0x7a67c92 0x704540f 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.288 INFO:teuthology.orchestra.run.trial091.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa0d2a157 0xffffffffa09f9006 0xffffffffa09fc9c0 0xffffffffa09fdd21 0xffffffffa0a25a53 0xffffffffa0a25c15 0xffffffffa09cc8ce 0xffffffffa09d4bab 0xffffffffa09d500b 0xffffffffa06910d2 0xffffffffa1350451 0xffffffffa1400ba2 2026-02-25T08:40:49.313 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a689f4 0x7d56cf3 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:49.352 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a695b0 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.353 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a686c6 0x7d56cde 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.353 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a69683 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.476 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcafa /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x157c7 0x9edcdd6 0x9edcf47 0x9edd1b7 0x9edd2bc 0x9f0f6c7 0x9f0fe6a 0x9f1a337 0x78b69db 0x7ba17e8 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.476 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xcde7a4 0x9e7862d 0x9ef56fa 0x9ef6a80 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.476 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xe99a /lib64/libasan.so.8+0xec776 0xcc26c1 0x7ba1959 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.477 INFO:teuthology.orchestra.run.trial091.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa1363497 0xffffffffa09e7985 0xffffffffa09cc97a 0xffffffffa09d4bab 0xffffffffa09d500b 0xffffffffa06910d2 0xffffffffa1350451 0xffffffffa1400ba2 2026-02-25T08:40:49.558 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a15220 0x2a69674 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:49.575 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a1526e 0x2a69674 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.575 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a68a31 0x7d56cf3 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:49.580 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a1526e 0x2a69674 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:51.749 INFO:teuthology.orchestra.run.trial091.stderr:----------------------------------------------------- 2026-02-25T08:40:51.749 INFO:teuthology.orchestra.run.trial091.stderr:Suppressions used: 2026-02-25T08:40:51.749 INFO:teuthology.orchestra.run.trial091.stderr: count bytes template 2026-02-25T08:40:51.750 INFO:teuthology.orchestra.run.trial091.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:51.750 INFO:teuthology.orchestra.run.trial091.stderr:----------------------------------------------------- 2026-02-25T08:40:51.750 INFO:teuthology.orchestra.run.trial091.stderr: 2026-02-25T08:40:51.786 DEBUG:teuthology.orchestra.run.trial091:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-25T08:40:51.811 DEBUG:teuthology.orchestra.run.trial091:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:51.963 INFO:teuthology.orchestra.run.trial091.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:52.015 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:52,015 seastar - Reactor backend: linux-aio 2026-02-25T08:40:52.029 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:52,029 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:52.029 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:52,029 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:52.031 INFO:teuthology.orchestra.run.trial091.stderr:==39146==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:52.087 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:52,087 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:52.088 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:52,087 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:52.088 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:52,087 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:52.088 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:52,087 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:40:52.088 INFO:teuthology.orchestra.run.trial091.stderr:DEBUG 2026-02-25 08:40:52,088 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:54.152 INFO:teuthology.orchestra.run.trial091.stderr:----------------------------------------------------- 2026-02-25T08:40:54.153 INFO:teuthology.orchestra.run.trial091.stderr:Suppressions used: 2026-02-25T08:40:54.153 INFO:teuthology.orchestra.run.trial091.stderr: count bytes template 2026-02-25T08:40:54.153 INFO:teuthology.orchestra.run.trial091.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:54.153 INFO:teuthology.orchestra.run.trial091.stderr:----------------------------------------------------- 2026-02-25T08:40:54.153 INFO:teuthology.orchestra.run.trial091.stderr: 2026-02-25T08:40:54.172 INFO:teuthology.orchestra.run.trial091.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:54.172 INFO:teuthology.orchestra.run.trial091.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:54.220 INFO:teuthology.orchestra.run.trial091.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:54.220 INFO:teuthology.orchestra.run.trial091.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:54.227 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:54,227 seastar - Reactor backend: linux-aio 2026-02-25T08:40:54.243 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:54,243 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:54.243 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:54,243 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:54.247 INFO:teuthology.orchestra.run.trial091.stderr:==39144==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:54.247 INFO:teuthology.orchestra.run.trial091.stderr:INFO 2026-02-25 08:40:54,247 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:54.307 INFO:teuthology.orchestra.run.trial091.stderr:DEBUG 2026-02-25 08:40:54,307 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:54.392 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xf0e77 /lib64/libasan.so.8+0xc9e9 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9cf61eb 0x9d73986 0x9d73a9c 0x9d73b78 0x9d73df4 0x9d73ead 0x9d7439b 0x9d7479f 0x9d74a5c 0x9f165e1 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.393 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15eeb 0x9e21a6a 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.393 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x10a26bf 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9f0751a 0x9f0916c 0x9f097c6 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.425 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xcc2834 0x10a2700 0xf47938 0x9f04791 0x9f04ad6 0x9f04e59 0x9f14fb0 0x9f18413 0x9ed89e1 0x9ed79e1 0x7b90d7c 0x7ba55d5 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:54.469 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 66 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x10b6c3 0x3610b77 0x6391195 0x7f691cb 0x7f694a2 0x7f695ce 0x7f6c143 0x7f6cd97 0x7f6d058 0x7e5013f 0x7fb8d74 0x9121733 0x91240a1 0x7857565 0x7a67c92 0x704540f 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.469 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x119128 /lib64/libasan.so.8+0xc5f4 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0xbca1d1 0xbca3da 0xbca917 0xd07b22 0x190ae94 0x9ef6afb 0x9ef6ea0 0x9f1568c 0x9f18413 0x9ed89e1 0x9ed79e1 0x7f64059 0x7fb93c0 0x9121733 0x91240a1 0x7857565 0x7a67c92 0x704540f 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.470 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 67 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9dd7048 0x9ed97cf 0x9ed9c5c 0x9f1a7dc 0x7c75b94 0x7ba71c8 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.493 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:54.534 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a69068 0x7d56d00 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.535 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x2a1526e 0x2a69674 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.537 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.667 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xea79b 0xcf0980 0xe2f26d 0xe2f42e 0x9ef6eb8 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.667 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xd077bd 0xff586f 0x9d756f4 0x9eec474 0x9eec56d 0x7ba1825 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.667 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc589 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9cf61eb 0x9d73986 0x9d73a9c 0x9d73b78 0x9d73df4 0x9d7439b 0x9d7479f 0x9d74a5c 0x9f165e1 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.667 INFO:teuthology.orchestra.run.trial091.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa0690e70 0xffffffffa06910bb 0xffffffffa1350451 0xffffffffa1400ba2 2026-02-25T08:40:54.749 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa14b388 0x9dc9011 0x9dc984c 0xefd639 /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58474 2026-02-25T08:40:54.766 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129747 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.766 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:54.767 INFO:teuthology.orchestra.run.trial091.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6592 /lib64/libasan.so.8+0x15197 0x2a696d3 0x7d56cc5 0x7e509f0 0x7e51257 0x7e4eef9 0x785215d 0x791938e 0x792debd 0x792e01c 0x79715a5 0x7971723 0x797199c 0x7aad4d2 0x791a321 0x791a39c 0x7934952 0x7a8e989 0x7a8ea18 0x7a8eabc 0x754f8ce 0x744618d 0x7510b35 0x760171a 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:56.951 INFO:teuthology.orchestra.run.trial091.stderr:----------------------------------------------------- 2026-02-25T08:40:56.951 INFO:teuthology.orchestra.run.trial091.stderr:Suppressions used: 2026-02-25T08:40:56.951 INFO:teuthology.orchestra.run.trial091.stderr: count bytes template 2026-02-25T08:40:56.951 INFO:teuthology.orchestra.run.trial091.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:56.951 INFO:teuthology.orchestra.run.trial091.stderr:----------------------------------------------------- 2026-02-25T08:40:56.951 INFO:teuthology.orchestra.run.trial091.stderr: 2026-02-25T08:40:56.987 DEBUG:teuthology.orchestra.run.trial091:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-25T08:40:57.012 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-25T08:40:57.013 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:57.013 DEBUG:teuthology.orchestra.run.trial039:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-25T08:40:57.040 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:57.040 DEBUG:teuthology.orchestra.run.trial039:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-25T08:40:57.103 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:40:57.103 DEBUG:teuthology.orchestra.run.trial091:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-25T08:40:57.128 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:40:57.128 DEBUG:teuthology.orchestra.run.trial091:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-25T08:40:57.193 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:40:57.193 DEBUG:teuthology.orchestra.run.trial091:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-25T08:40:57.257 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:57.258 DEBUG:teuthology.orchestra.run.trial039:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-25T08:40:57.276 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:40:57.276 DEBUG:teuthology.orchestra.run.trial091:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2026-02-25T08:40:57.315 INFO:tasks.ceph:Adding keys to all mons... 2026-02-25T08:40:57.315 DEBUG:teuthology.orchestra.run.trial039:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-25T08:40:57.340 INFO:teuthology.orchestra.run.trial039.stdout:[osd.0] 2026-02-25T08:40:57.340 INFO:teuthology.orchestra.run.trial039.stdout: key = AQCFtZ5pYvn8CxAAxImoMJFcvrdJfXqbYojUkQ== 2026-02-25T08:40:57.340 INFO:teuthology.orchestra.run.trial039.stdout:[osd.1] 2026-02-25T08:40:57.340 INFO:teuthology.orchestra.run.trial039.stdout: key = AQCKtZ5pK1v4FhAArNROCwqNmdApdJsReegucQ== 2026-02-25T08:40:57.340 INFO:teuthology.orchestra.run.trial039.stdout:[mgr.x] 2026-02-25T08:40:57.340 INFO:teuthology.orchestra.run.trial039.stdout: key = AQCAtZ5pZZ2yOBAAM2fe2+GrAaKUOQJYsLQk+Q== 2026-02-25T08:40:57.340 INFO:teuthology.orchestra.run.trial039.stdout:[osd.2] 2026-02-25T08:40:57.341 INFO:teuthology.orchestra.run.trial039.stdout: key = AQCRtZ5pEhTABxAAx4TZtZhtPQNoVcxlzMSVKw== 2026-02-25T08:40:57.341 INFO:teuthology.orchestra.run.trial039.stdout:[osd.3] 2026-02-25T08:40:57.341 INFO:teuthology.orchestra.run.trial039.stdout: key = AQCWtZ5pbXeNEhAAVdk3/O6lU0TeSQH75fSTkA== 2026-02-25T08:40:57.341 INFO:teuthology.orchestra.run.trial039.stdout:[client.0] 2026-02-25T08:40:57.341 INFO:teuthology.orchestra.run.trial039.stdout: key = AQCAtZ5p4IZpOxAAklK9quPdDd0K+bRtEcUa4Q== 2026-02-25T08:40:57.341 INFO:teuthology.orchestra.run.trial039.stdout:[client.1] 2026-02-25T08:40:57.341 INFO:teuthology.orchestra.run.trial039.stdout: key = AQCBtZ5ps9YeAxAALSeFbAla3rv6evxOmP0lvg== 2026-02-25T08:40:57.342 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:40:57.430 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:40:57.478 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-25T08:40:57.526 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:40:57.574 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:40:57.622 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T08:40:57.671 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T08:40:57.720 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-25T08:40:57.720 DEBUG:teuthology.orchestra.run.trial039:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-25T08:40:57.745 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-25T08:40:57.830 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:57.829+0000 7fb8ce233c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:57.831 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:57.829+0000 7fb8ce233c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:57.842 DEBUG:teuthology.orchestra.run.trial039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-25T08:40:57.867 DEBUG:teuthology.orchestra.run.trial039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:57.923 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-25T08:40:57.924 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-25T08:40:57.924 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-25T08:40:57.965 INFO:tasks.ceph.mon.a:Started 2026-02-25T08:40:57.966 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-25T08:40:57.966 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-25T08:40:57.966 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-25T08:40:57.969 INFO:tasks.ceph.mgr.x:Started 2026-02-25T08:40:57.969 DEBUG:tasks.ceph:set 0 configs 2026-02-25T08:40:57.969 DEBUG:teuthology.orchestra.run.trial039:> sudo ceph --cluster ceph config dump 2026-02-25T08:40:58.027 INFO:tasks.ceph.mon.a.trial039.stderr:2026-02-25T08:40:58.025+0000 7f2b84c3ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.027 INFO:tasks.ceph.mon.a.trial039.stderr:2026-02-25T08:40:58.025+0000 7f2b84c3ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.027 INFO:tasks.ceph.mon.a.trial039.stderr:2026-02-25T08:40:58.025+0000 7f2b84c3ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.028 INFO:tasks.ceph.mgr.x.trial091.stderr:2026-02-25T08:40:58.028+0000 7fcf3642efc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.045 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:58.041+0000 7f654d5b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.048 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:58.047+0000 7f654d5b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.081 INFO:teuthology.orchestra.run.trial039.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-25T08:40:58.089 INFO:tasks.ceph:Setting crush tunables to default 2026-02-25T08:40:58.090 DEBUG:teuthology.orchestra.run.trial039:> sudo ceph --cluster ceph osd crush tunables default 2026-02-25T08:40:58.143 INFO:tasks.ceph.mgr.x.trial091.stderr:2026-02-25T08:40:58.143+0000 7fcf3642efc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.144 INFO:tasks.ceph.mgr.x.trial091.stderr:2026-02-25T08:40:58.143+0000 7fcf3642efc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.164 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:58.160+0000 7f91934b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.167 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:58.166+0000 7f91934b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.203 INFO:teuthology.orchestra.run.trial039.stderr:adjusted tunables profile to default 2026-02-25T08:40:58.211 INFO:tasks.ceph:check_enable_crimson: True 2026-02-25T08:40:58.211 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-25T08:40:58.211 DEBUG:teuthology.orchestra.run.trial039:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-25T08:40:58.286 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:58.282+0000 7f6fe5046640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.289 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:40:58.288+0000 7f6fe5046640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:58.328 INFO:tasks.ceph.mon.a.trial039.stderr:2026-02-25T08:40:58.327+0000 7f2b7bff1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-25T08:40:58.328 INFO:tasks.ceph.mon.a.trial039.stderr:Please be aware that this feature is experimental, untested, 2026-02-25T08:40:58.329 INFO:tasks.ceph.mon.a.trial039.stderr:unsupported, and may result in data corruption, data loss, 2026-02-25T08:40:58.329 INFO:tasks.ceph.mon.a.trial039.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-25T08:40:58.329 INFO:tasks.ceph.mon.a.trial039.stderr:feature with important data. 2026-02-25T08:40:58.329 INFO:tasks.ceph.mon.a.trial039.stderr: 2026-02-25T08:40:58.986 INFO:tasks.ceph.mgr.x.trial091.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-25T08:40:58.986 INFO:tasks.ceph.mgr.x.trial091.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-25T08:40:58.986 INFO:tasks.ceph.mgr.x.trial091.stderr: from numpy import show_config as show_numpy_config 2026-02-25T08:40:59.214 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-25T08:40:59.215 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:59.215 DEBUG:teuthology.orchestra.run.trial039:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-25T08:40:59.240 DEBUG:teuthology.orchestra.run.trial039:> set -ex 2026-02-25T08:40:59.240 DEBUG:teuthology.orchestra.run.trial039:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-25T08:40:59.306 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:40:59.306 DEBUG:teuthology.orchestra.run.trial091:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-25T08:40:59.331 DEBUG:teuthology.orchestra.run.trial091:> set -ex 2026-02-25T08:40:59.331 DEBUG:teuthology.orchestra.run.trial091:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-25T08:40:59.396 DEBUG:teuthology.orchestra.run.trial091:> sudo ceph --cluster ceph osd new 529301d2-73dd-4079-88ca-4064a8becb6e 0 2026-02-25T08:40:59.509 INFO:teuthology.orchestra.run.trial091.stderr:2026-02-25T08:40:59.506+0000 7fbdfaecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:59.512 INFO:teuthology.orchestra.run.trial091.stderr:2026-02-25T08:40:59.512+0000 7fbdfaecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:59.550 INFO:teuthology.orchestra.run.trial091.stdout:0 2026-02-25T08:40:59.558 DEBUG:teuthology.orchestra.run.trial091:> sudo ceph --cluster ceph osd new fcf3ee73-be1b-484d-a5a7-2aee2fa8c451 1 2026-02-25T08:40:59.631 INFO:teuthology.orchestra.run.trial091.stderr:2026-02-25T08:40:59.627+0000 7f1d5c45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:59.633 INFO:teuthology.orchestra.run.trial091.stderr:2026-02-25T08:40:59.632+0000 7f1d5c45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:59.670 INFO:teuthology.orchestra.run.trial091.stdout:1 2026-02-25T08:40:59.677 DEBUG:teuthology.orchestra.run.trial091:> sudo ceph --cluster ceph osd new 8b58ba7d-53ad-46af-9b4f-bd4f6994d8b0 2 2026-02-25T08:40:59.751 INFO:teuthology.orchestra.run.trial091.stderr:2026-02-25T08:40:59.747+0000 7f6ef9c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:59.753 INFO:teuthology.orchestra.run.trial091.stderr:2026-02-25T08:40:59.752+0000 7f6ef9c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:59.791 INFO:teuthology.orchestra.run.trial091.stdout:2 2026-02-25T08:40:59.799 DEBUG:teuthology.orchestra.run.trial091:> sudo ceph --cluster ceph osd new f5427ebf-811a-4afc-8c54-a422f25ec213 3 2026-02-25T08:40:59.873 INFO:teuthology.orchestra.run.trial091.stderr:2026-02-25T08:40:59.870+0000 7fa983a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:59.876 INFO:teuthology.orchestra.run.trial091.stderr:2026-02-25T08:40:59.876+0000 7fa983a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:59.914 INFO:teuthology.orchestra.run.trial091.stdout:3 2026-02-25T08:40:59.923 DEBUG:teuthology.orchestra.run.trial039:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-25T08:40:59.952 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-25T08:40:59.952 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-25T08:40:59.994 INFO:tasks.ceph.osd.0:Started 2026-02-25T08:40:59.995 DEBUG:teuthology.orchestra.run.trial039:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-25T08:41:00.023 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-25T08:41:00.023 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-25T08:41:00.064 INFO:tasks.ceph.osd.1:Started 2026-02-25T08:41:00.065 DEBUG:teuthology.orchestra.run.trial091:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-25T08:41:00.093 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-25T08:41:00.093 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-25T08:41:00.134 INFO:tasks.ceph.osd.2:Started 2026-02-25T08:41:00.135 DEBUG:teuthology.orchestra.run.trial091:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2026-02-25T08:41:00.164 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-25T08:41:00.164 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2026-02-25T08:41:00.206 INFO:tasks.ceph.osd.3:Started 2026-02-25T08:41:00.207 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:00.282 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:00.279+0000 7fddc82cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:00.285 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:00.284+0000 7fddc82cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:00.319 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:41:00.319 INFO:teuthology.orchestra.run.trial039.stdout:{"epoch":7,"fsid":"10023e6d-f681-4f84-90aa-0a6f37c6d62c","created":"2026-02-25T08:40:58.038353+0000","modified":"2026-02-25T08:40:59.913808+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:40:59.913808+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"529301d2-73dd-4079-88ca-4064a8becb6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fcf3ee73-be1b-484d-a5a7-2aee2fa8c451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8b58ba7d-53ad-46af-9b4f-bd4f6994d8b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f5427ebf-811a-4afc-8c54-a422f25ec213","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:00.327 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-25T08:41:00.327 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-25T08:41:00.729 DEBUG:teuthology.orchestra.run.trial039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:00.797 INFO:teuthology.misc.health.trial039.stderr:2026-02-25T08:41:00.793+0000 7f504d374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:00.799 INFO:teuthology.misc.health.trial039.stderr:2026-02-25T08:41:00.798+0000 7f504d374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:00.832 INFO:teuthology.misc.health.trial039.stdout: 2026-02-25T08:41:00.832 INFO:teuthology.misc.health.trial039.stdout:{"epoch":7,"fsid":"10023e6d-f681-4f84-90aa-0a6f37c6d62c","created":"2026-02-25T08:40:58.038353+0000","modified":"2026-02-25T08:40:59.913808+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:40:59.913808+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"529301d2-73dd-4079-88ca-4064a8becb6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fcf3ee73-be1b-484d-a5a7-2aee2fa8c451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8b58ba7d-53ad-46af-9b4f-bd4f6994d8b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f5427ebf-811a-4afc-8c54-a422f25ec213","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:00.838 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-25T08:41:01.378 INFO:tasks.ceph.mgr.x.trial091.stderr:2026-02-25T08:41:01.378+0000 7fce5185f640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-25T08:41:07.241 DEBUG:teuthology.orchestra.run.trial039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:07.311 INFO:teuthology.misc.health.trial039.stderr:2026-02-25T08:41:07.307+0000 7fb6d0c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:07.313 INFO:teuthology.misc.health.trial039.stderr:2026-02-25T08:41:07.312+0000 7fb6d0c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:07.427 INFO:teuthology.misc.health.trial039.stdout: 2026-02-25T08:41:07.428 INFO:teuthology.misc.health.trial039.stdout:{"epoch":10,"fsid":"10023e6d-f681-4f84-90aa-0a6f37c6d62c","created":"2026-02-25T08:40:58.038353+0000","modified":"2026-02-25T08:41:06.540862+0000","last_up_change":"2026-02-25T08:41:06.540862+0000","last_in_change":"2026-02-25T08:40:59.913808+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"529301d2-73dd-4079-88ca-4064a8becb6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"fcf3ee73-be1b-484d-a5a7-2aee2fa8c451","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8b58ba7d-53ad-46af-9b4f-bd4f6994d8b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f5427ebf-811a-4afc-8c54-a422f25ec213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6803","nonce":3250748496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6802","nonce":3250748496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6808","nonce":3250748496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6807","nonce":3250748496}]},"public_addr":"10.20.193.91:6803/3250748496","cluster_addr":"10.20.193.91:6802/3250748496","heartbeat_back_addr":"10.20.193.91:6808/3250748496","heartbeat_front_addr":"10.20.193.91:6807/3250748496","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:07.436 DEBUG:teuthology.misc:1 of 4 OSDs are up 2026-02-25T08:41:13.839 DEBUG:teuthology.orchestra.run.trial039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:13.909 INFO:teuthology.misc.health.trial039.stderr:2026-02-25T08:41:13.904+0000 7efd1d70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.911 INFO:teuthology.misc.health.trial039.stderr:2026-02-25T08:41:13.909+0000 7efd1d70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:14.025 INFO:teuthology.misc.health.trial039.stdout: 2026-02-25T08:41:14.025 INFO:teuthology.misc.health.trial039.stdout:{"epoch":14,"fsid":"10023e6d-f681-4f84-90aa-0a6f37c6d62c","created":"2026-02-25T08:40:58.038353+0000","modified":"2026-02-25T08:41:10.558237+0000","last_up_change":"2026-02-25T08:41:07.545934+0000","last_in_change":"2026-02-25T08:40:59.913808+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-25T08:41:08.394581+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"529301d2-73dd-4079-88ca-4064a8becb6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6803","nonce":1465499150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6800","nonce":1465499150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6806","nonce":1465499150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6808","nonce":1465499150}]},"public_addr":"10.20.193.39:6803/1465499150","cluster_addr":"10.20.193.39:6800/1465499150","heartbeat_back_addr":"10.20.193.39:6806/1465499150","heartbeat_front_addr":"10.20.193.39:6808/1465499150","state":["exists","up"]},{"osd":1,"uuid":"fcf3ee73-be1b-484d-a5a7-2aee2fa8c451","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6802","nonce":148626133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6801","nonce":148626133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6804","nonce":148626133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6807","nonce":148626133}]},"public_addr":"10.20.193.39:6802/148626133","cluster_addr":"10.20.193.39:6801/148626133","heartbeat_back_addr":"10.20.193.39:6804/148626133","heartbeat_front_addr":"10.20.193.39:6807/148626133","state":["exists","up"]},{"osd":2,"uuid":"8b58ba7d-53ad-46af-9b4f-bd4f6994d8b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6805","nonce":3625174827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6806","nonce":3625174827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6810","nonce":3625174827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6811","nonce":3625174827}]},"public_addr":"10.20.193.91:6805/3625174827","cluster_addr":"10.20.193.91:6806/3625174827","heartbeat_back_addr":"10.20.193.91:6810/3625174827","heartbeat_front_addr":"10.20.193.91:6811/3625174827","state":["exists","up"]},{"osd":3,"uuid":"f5427ebf-811a-4afc-8c54-a422f25ec213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6803","nonce":3250748496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6802","nonce":3250748496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6808","nonce":3250748496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6807","nonce":3250748496}]},"public_addr":"10.20.193.91:6803/3250748496","cluster_addr":"10.20.193.91:6802/3250748496","heartbeat_back_addr":"10.20.193.91:6808/3250748496","heartbeat_front_addr":"10.20.193.91:6807/3250748496","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:14.035 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-02-25T08:41:14.035 INFO:tasks.ceph:Creating RBD pool 2026-02-25T08:41:14.035 DEBUG:teuthology.orchestra.run.trial039:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-25T08:41:14.110 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:14.103+0000 7fc8dac8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:14.112 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:14.110+0000 7fc8dac8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:15.080 INFO:teuthology.orchestra.run.trial039.stderr:pool 'rbd' created 2026-02-25T08:41:15.094 DEBUG:teuthology.orchestra.run.trial039:> rbd --cluster ceph pool init rbd 2026-02-25T08:41:15.139 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:15.136+0000 7ff96cee4240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:15.139 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:15.136+0000 7ff96cee4240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:15.142 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:15.140+0000 7ff96cee4240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.117 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-25T08:41:18.117 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-25T08:41:18.118 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-25T08:41:18.194 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:18.188+0000 7fad0ef0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.196 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:18.194+0000 7fad0ef0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.308 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:41:18.317 INFO:teuthology.orchestra.run.trial039.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-25T08:41:18.317 INFO:tasks.ceph_manager:config epoch is 1 2026-02-25T08:41:18.317 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-25T08:41:18.317 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-25T08:41:18.318 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-25T08:41:18.394 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:18.390+0000 7fe46fe1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.397 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:18.394+0000 7fe46fe1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.541 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:41:18.549 INFO:teuthology.orchestra.run.trial039.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6800","nonce":709555552},{"type":"v1","addr":"10.20.193.91:6801","nonce":709555552}]},"active_addr":"10.20.193.91:6801/709555552","active_change":"2026-02-25T08:41:00.371419+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.91:0","nonce":2594775128}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.91:0","nonce":3839238411}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.91:0","nonce":4218946985}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.91:0","nonce":3677699324}]}]} 2026-02-25T08:41:18.550 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-25T08:41:18.551 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T08:41:18.551 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:18.625 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:18.620+0000 7f59941dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.627 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:18.625+0000 7f59941dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.743 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:41:18.743 INFO:teuthology.orchestra.run.trial039.stdout:{"epoch":18,"fsid":"10023e6d-f681-4f84-90aa-0a6f37c6d62c","created":"2026-02-25T08:40:58.038353+0000","modified":"2026-02-25T08:41:18.088268+0000","last_up_change":"2026-02-25T08:41:07.545934+0000","last_in_change":"2026-02-25T08:40:59.913808+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-25T08:41:08.394581+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-25T08:41:14.225857+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"529301d2-73dd-4079-88ca-4064a8becb6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6803","nonce":1465499150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6800","nonce":1465499150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6806","nonce":1465499150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6808","nonce":1465499150}]},"public_addr":"10.20.193.39:6803/1465499150","cluster_addr":"10.20.193.39:6800/1465499150","heartbeat_back_addr":"10.20.193.39:6806/1465499150","heartbeat_front_addr":"10.20.193.39:6808/1465499150","state":["exists","up"]},{"osd":1,"uuid":"fcf3ee73-be1b-484d-a5a7-2aee2fa8c451","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6802","nonce":148626133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6801","nonce":148626133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6804","nonce":148626133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6807","nonce":148626133}]},"public_addr":"10.20.193.39:6802/148626133","cluster_addr":"10.20.193.39:6801/148626133","heartbeat_back_addr":"10.20.193.39:6804/148626133","heartbeat_front_addr":"10.20.193.39:6807/148626133","state":["exists","up"]},{"osd":2,"uuid":"8b58ba7d-53ad-46af-9b4f-bd4f6994d8b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6805","nonce":3625174827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6806","nonce":3625174827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6810","nonce":3625174827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6811","nonce":3625174827}]},"public_addr":"10.20.193.91:6805/3625174827","cluster_addr":"10.20.193.91:6806/3625174827","heartbeat_back_addr":"10.20.193.91:6810/3625174827","heartbeat_front_addr":"10.20.193.91:6811/3625174827","state":["exists","up"]},{"osd":3,"uuid":"f5427ebf-811a-4afc-8c54-a422f25ec213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6803","nonce":3250748496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6802","nonce":3250748496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6808","nonce":3250748496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6807","nonce":3250748496}]},"public_addr":"10.20.193.91:6803/3250748496","cluster_addr":"10.20.193.91:6802/3250748496","heartbeat_back_addr":"10.20.193.91:6808/3250748496","heartbeat_front_addr":"10.20.193.91:6807/3250748496","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:18.753 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-25T08:41:18.753 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:18.830 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:18.826+0000 7fc1beec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.832 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:18.830+0000 7fc1beec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:18.946 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:41:18.947 INFO:teuthology.orchestra.run.trial039.stdout:{"epoch":18,"fsid":"10023e6d-f681-4f84-90aa-0a6f37c6d62c","created":"2026-02-25T08:40:58.038353+0000","modified":"2026-02-25T08:41:18.088268+0000","last_up_change":"2026-02-25T08:41:07.545934+0000","last_in_change":"2026-02-25T08:40:59.913808+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-25T08:41:08.394581+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-25T08:41:14.225857+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"529301d2-73dd-4079-88ca-4064a8becb6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6803","nonce":1465499150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6800","nonce":1465499150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6806","nonce":1465499150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6808","nonce":1465499150}]},"public_addr":"10.20.193.39:6803/1465499150","cluster_addr":"10.20.193.39:6800/1465499150","heartbeat_back_addr":"10.20.193.39:6806/1465499150","heartbeat_front_addr":"10.20.193.39:6808/1465499150","state":["exists","up"]},{"osd":1,"uuid":"fcf3ee73-be1b-484d-a5a7-2aee2fa8c451","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6802","nonce":148626133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6801","nonce":148626133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6804","nonce":148626133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.39:6807","nonce":148626133}]},"public_addr":"10.20.193.39:6802/148626133","cluster_addr":"10.20.193.39:6801/148626133","heartbeat_back_addr":"10.20.193.39:6804/148626133","heartbeat_front_addr":"10.20.193.39:6807/148626133","state":["exists","up"]},{"osd":2,"uuid":"8b58ba7d-53ad-46af-9b4f-bd4f6994d8b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6805","nonce":3625174827}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6806","nonce":3625174827}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6810","nonce":3625174827}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6811","nonce":3625174827}]},"public_addr":"10.20.193.91:6805/3625174827","cluster_addr":"10.20.193.91:6806/3625174827","heartbeat_back_addr":"10.20.193.91:6810/3625174827","heartbeat_front_addr":"10.20.193.91:6811/3625174827","state":["exists","up"]},{"osd":3,"uuid":"f5427ebf-811a-4afc-8c54-a422f25ec213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6803","nonce":3250748496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6802","nonce":3250748496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6808","nonce":3250748496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.91:6807","nonce":3250748496}]},"public_addr":"10.20.193.91:6803/3250748496","cluster_addr":"10.20.193.91:6802/3250748496","heartbeat_back_addr":"10.20.193.91:6808/3250748496","heartbeat_front_addr":"10.20.193.91:6807/3250748496","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:18.958 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-25T08:41:18.958 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-25T08:41:18.959 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-25T08:41:18.959 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-25T08:41:19.041 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.036+0000 7f83c8f8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.042 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.036+0000 7f6c7690b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.042 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.036+0000 7f3b97b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.042 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.036+0000 7f1064ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.043 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.041+0000 7f83c8f8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.044 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.042+0000 7f3b97b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.045 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.042+0000 7f6c7690b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.045 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.043+0000 7f1064ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.061 INFO:teuthology.orchestra.run.trial039.stdout:47244640267 2026-02-25T08:41:19.061 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:41:19.063 INFO:teuthology.orchestra.run.trial039.stdout:47244640267 2026-02-25T08:41:19.063 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T08:41:19.064 INFO:teuthology.orchestra.run.trial039.stdout:47244640267 2026-02-25T08:41:19.065 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:41:19.065 INFO:teuthology.orchestra.run.trial039.stdout:42949672972 2026-02-25T08:41:19.066 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-25T08:41:19.140 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.134+0000 7f4d8a85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.143 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.140+0000 7f4d8a85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.186 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.181+0000 7f8ba63dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.187 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.181+0000 7f2d04ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.187 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.181+0000 7fd5fc8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.189 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.187+0000 7f8ba63dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.189 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.187+0000 7fd5fc8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.189 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:19.187+0000 7f2d04ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:19.258 INFO:teuthology.orchestra.run.trial039.stdout:47244640266 2026-02-25T08:41:19.270 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640266 for osd.1 2026-02-25T08:41:19.302 INFO:teuthology.orchestra.run.trial039.stdout:42949672969 2026-02-25T08:41:19.306 INFO:teuthology.orchestra.run.trial039.stdout:47244640266 2026-02-25T08:41:19.309 INFO:teuthology.orchestra.run.trial039.stdout:47244640266 2026-02-25T08:41:19.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672972 got 42949672969 for osd.3 2026-02-25T08:41:19.317 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640266 for osd.2 2026-02-25T08:41:19.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640266 for osd.0 2026-02-25T08:41:20.271 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:41:20.314 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-25T08:41:20.318 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:41:20.322 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T08:41:20.351 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:20.346+0000 7ff71a3f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.353 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:20.351+0000 7ff71a3f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.391 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:20.386+0000 7f138efd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.393 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:20.391+0000 7f138efd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.429 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:20.424+0000 7f279e562640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.430 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:20.425+0000 7f4db6ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.432 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:20.430+0000 7f4db6ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.432 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:20.430+0000 7f279e562640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:20.469 INFO:teuthology.orchestra.run.trial039.stdout:47244640266 2026-02-25T08:41:20.480 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640266 for osd.1 2026-02-25T08:41:20.508 INFO:teuthology.orchestra.run.trial039.stdout:42949672969 2026-02-25T08:41:20.519 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672972 got 42949672969 for osd.3 2026-02-25T08:41:20.545 INFO:teuthology.orchestra.run.trial039.stdout:47244640266 2026-02-25T08:41:20.550 INFO:teuthology.orchestra.run.trial039.stdout:47244640266 2026-02-25T08:41:20.556 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640266 for osd.2 2026-02-25T08:41:20.561 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640266 for osd.0 2026-02-25T08:41:21.481 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:41:21.520 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-25T08:41:21.557 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:41:21.559 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:21.554+0000 7f3b89ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:21.562 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T08:41:21.563 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:21.560+0000 7f3b89ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:21.602 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:21.596+0000 7f01026cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:21.604 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:21.602+0000 7f01026cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:21.637 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:21.633+0000 7fa4f29a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:21.638 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:21.634+0000 7f04afe5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:21.640 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:21.638+0000 7fa4f29a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:21.641 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:21.639+0000 7f04afe5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:21.678 INFO:teuthology.orchestra.run.trial039.stdout:47244640267 2026-02-25T08:41:21.689 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640267 for osd.1 2026-02-25T08:41:21.690 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:21.724 INFO:teuthology.orchestra.run.trial039.stdout:42949672972 2026-02-25T08:41:21.735 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672972 got 42949672972 for osd.3 2026-02-25T08:41:21.735 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:21.755 INFO:teuthology.orchestra.run.trial039.stdout:47244640267 2026-02-25T08:41:21.757 INFO:teuthology.orchestra.run.trial039.stdout:47244640267 2026-02-25T08:41:21.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640267 for osd.2 2026-02-25T08:41:21.767 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:21.769 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640267 for osd.0 2026-02-25T08:41:21.769 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:21.770 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:41:21.770 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:21.887 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:21.882+0000 7f2f9b5e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:21.889 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:21.887+0000 7f2f9b5e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.000 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:41:22.000 INFO:teuthology.orchestra.run.trial039.stderr:dumped all 2026-02-25T08:41:22.009 INFO:teuthology.orchestra.run.trial039.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-25T08:41:20.377063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4224,"kb_used_data":4224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662400,"statfs":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.296303"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.100814+0000","last_change":"2026-02-25T08:41:18.104814+0000","last_active":"2026-02-25T08:41:18.100814+0000","last_peered":"2026-02-25T08:41:18.100814+0000","last_clean":"2026-02-25T08:41:18.100814+0000","last_became_active":"2026-02-25T08:41:16.142826+0000","last_became_peered":"2026-02-25T08:41:16.142826+0000","last_unstale":"2026-02-25T08:41:18.100814+0000","last_undegraded":"2026-02-25T08:41:18.100814+0000","last_fullsized":"2026-02-25T08:41:18.100814+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101069+0000","last_change":"2026-02-25T08:41:18.103612+0000","last_active":"2026-02-25T08:41:18.101069+0000","last_peered":"2026-02-25T08:41:18.101069+0000","last_clean":"2026-02-25T08:41:18.101069+0000","last_became_active":"2026-02-25T08:41:16.124173+0000","last_became_peered":"2026-02-25T08:41:16.124173+0000","last_unstale":"2026-02-25T08:41:18.101069+0000","last_undegraded":"2026-02-25T08:41:18.101069+0000","last_fullsized":"2026-02-25T08:41:18.101069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101938+0000","last_change":"2026-02-25T08:41:18.105179+0000","last_active":"2026-02-25T08:41:18.101938+0000","last_peered":"2026-02-25T08:41:18.101938+0000","last_clean":"2026-02-25T08:41:18.101938+0000","last_became_active":"2026-02-25T08:41:16.123155+0000","last_became_peered":"2026-02-25T08:41:16.123155+0000","last_unstale":"2026-02-25T08:41:18.101938+0000","last_undegraded":"2026-02-25T08:41:18.101938+0000","last_fullsized":"2026-02-25T08:41:18.101938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103444+0000","last_change":"2026-02-25T08:41:18.105717+0000","last_active":"2026-02-25T08:41:18.103444+0000","last_peered":"2026-02-25T08:41:18.103444+0000","last_clean":"2026-02-25T08:41:18.103444+0000","last_became_active":"2026-02-25T08:41:16.123587+0000","last_became_peered":"2026-02-25T08:41:16.123587+0000","last_unstale":"2026-02-25T08:41:18.103444+0000","last_undegraded":"2026-02-25T08:41:18.103444+0000","last_fullsized":"2026-02-25T08:41:18.103444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'1","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103630+0000","last_change":"2026-02-25T08:41:18.103630+0000","last_active":"2026-02-25T08:41:18.103630+0000","last_peered":"2026-02-25T08:41:18.103630+0000","last_clean":"2026-02-25T08:41:18.103630+0000","last_became_active":"2026-02-25T08:41:16.124400+0000","last_became_peered":"2026-02-25T08:41:16.124400+0000","last_unstale":"2026-02-25T08:41:18.103630+0000","last_undegraded":"2026-02-25T08:41:18.103630+0000","last_fullsized":"2026-02-25T08:41:18.103630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103342+0000","last_change":"2026-02-25T08:41:18.105717+0000","last_active":"2026-02-25T08:41:18.103342+0000","last_peered":"2026-02-25T08:41:18.103342+0000","last_clean":"2026-02-25T08:41:18.103342+0000","last_became_active":"2026-02-25T08:41:16.127216+0000","last_became_peered":"2026-02-25T08:41:16.127216+0000","last_unstale":"2026-02-25T08:41:18.103342+0000","last_undegraded":"2026-02-25T08:41:18.103342+0000","last_fullsized":"2026-02-25T08:41:18.103342+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101098+0000","last_change":"2026-02-25T08:41:18.103749+0000","last_active":"2026-02-25T08:41:18.101098+0000","last_peered":"2026-02-25T08:41:18.101098+0000","last_clean":"2026-02-25T08:41:18.101098+0000","last_became_active":"2026-02-25T08:41:16.125155+0000","last_became_peered":"2026-02-25T08:41:16.125155+0000","last_unstale":"2026-02-25T08:41:18.101098+0000","last_undegraded":"2026-02-25T08:41:18.101098+0000","last_fullsized":"2026-02-25T08:41:18.101098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101259+0000","last_change":"2026-02-25T08:41:18.103989+0000","last_active":"2026-02-25T08:41:18.101259+0000","last_peered":"2026-02-25T08:41:18.101259+0000","last_clean":"2026-02-25T08:41:18.101259+0000","last_became_active":"2026-02-25T08:41:16.133525+0000","last_became_peered":"2026-02-25T08:41:16.133525+0000","last_unstale":"2026-02-25T08:41:18.101259+0000","last_undegraded":"2026-02-25T08:41:18.101259+0000","last_fullsized":"2026-02-25T08:41:18.101259+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":91,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.100204+0000","last_change":"2026-02-25T08:41:09.603704+0000","last_active":"2026-02-25T08:41:18.100204+0000","last_peered":"2026-02-25T08:41:18.100204+0000","last_clean":"2026-02-25T08:41:18.100204+0000","last_became_active":"2026-02-25T08:41:09.591425+0000","last_became_peered":"2026-02-25T08:41:09.591425+0000","last_unstale":"2026-02-25T08:41:18.100204+0000","last_undegraded":"2026-02-25T08:41:18.100204+0000","last_fullsized":"2026-02-25T08:41:18.100204+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:08.549571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:08.549571+0000","last_clean_scrub_stamp":"2026-02-25T08:41:08.549571+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160252022784,"internally_reserved":0,"allocated":3944448,"data_stored":3944448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672972,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":372,"kb_used_data":372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166284,"statfs":{"total":53418655744,"available":53418274816,"internally_reserved":0,"allocated":380928,"data_stored":380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165324,"statfs":{"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1276,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165380,"statfs":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1244,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165412,"statfs":{"total":53418655744,"available":53417381888,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417381888,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:22.011 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:22.088 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:22.083+0000 7f740e6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.090 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:22.088+0000 7f740e6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.200 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:41:22.201 INFO:teuthology.orchestra.run.trial039.stderr:dumped all 2026-02-25T08:41:22.210 INFO:teuthology.orchestra.run.trial039.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-25T08:41:20.377063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4224,"kb_used_data":4224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662400,"statfs":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.296303"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.100814+0000","last_change":"2026-02-25T08:41:18.104814+0000","last_active":"2026-02-25T08:41:18.100814+0000","last_peered":"2026-02-25T08:41:18.100814+0000","last_clean":"2026-02-25T08:41:18.100814+0000","last_became_active":"2026-02-25T08:41:16.142826+0000","last_became_peered":"2026-02-25T08:41:16.142826+0000","last_unstale":"2026-02-25T08:41:18.100814+0000","last_undegraded":"2026-02-25T08:41:18.100814+0000","last_fullsized":"2026-02-25T08:41:18.100814+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101069+0000","last_change":"2026-02-25T08:41:18.103612+0000","last_active":"2026-02-25T08:41:18.101069+0000","last_peered":"2026-02-25T08:41:18.101069+0000","last_clean":"2026-02-25T08:41:18.101069+0000","last_became_active":"2026-02-25T08:41:16.124173+0000","last_became_peered":"2026-02-25T08:41:16.124173+0000","last_unstale":"2026-02-25T08:41:18.101069+0000","last_undegraded":"2026-02-25T08:41:18.101069+0000","last_fullsized":"2026-02-25T08:41:18.101069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101938+0000","last_change":"2026-02-25T08:41:18.105179+0000","last_active":"2026-02-25T08:41:18.101938+0000","last_peered":"2026-02-25T08:41:18.101938+0000","last_clean":"2026-02-25T08:41:18.101938+0000","last_became_active":"2026-02-25T08:41:16.123155+0000","last_became_peered":"2026-02-25T08:41:16.123155+0000","last_unstale":"2026-02-25T08:41:18.101938+0000","last_undegraded":"2026-02-25T08:41:18.101938+0000","last_fullsized":"2026-02-25T08:41:18.101938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103444+0000","last_change":"2026-02-25T08:41:18.105717+0000","last_active":"2026-02-25T08:41:18.103444+0000","last_peered":"2026-02-25T08:41:18.103444+0000","last_clean":"2026-02-25T08:41:18.103444+0000","last_became_active":"2026-02-25T08:41:16.123587+0000","last_became_peered":"2026-02-25T08:41:16.123587+0000","last_unstale":"2026-02-25T08:41:18.103444+0000","last_undegraded":"2026-02-25T08:41:18.103444+0000","last_fullsized":"2026-02-25T08:41:18.103444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'1","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103630+0000","last_change":"2026-02-25T08:41:18.103630+0000","last_active":"2026-02-25T08:41:18.103630+0000","last_peered":"2026-02-25T08:41:18.103630+0000","last_clean":"2026-02-25T08:41:18.103630+0000","last_became_active":"2026-02-25T08:41:16.124400+0000","last_became_peered":"2026-02-25T08:41:16.124400+0000","last_unstale":"2026-02-25T08:41:18.103630+0000","last_undegraded":"2026-02-25T08:41:18.103630+0000","last_fullsized":"2026-02-25T08:41:18.103630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103342+0000","last_change":"2026-02-25T08:41:18.105717+0000","last_active":"2026-02-25T08:41:18.103342+0000","last_peered":"2026-02-25T08:41:18.103342+0000","last_clean":"2026-02-25T08:41:18.103342+0000","last_became_active":"2026-02-25T08:41:16.127216+0000","last_became_peered":"2026-02-25T08:41:16.127216+0000","last_unstale":"2026-02-25T08:41:18.103342+0000","last_undegraded":"2026-02-25T08:41:18.103342+0000","last_fullsized":"2026-02-25T08:41:18.103342+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101098+0000","last_change":"2026-02-25T08:41:18.103749+0000","last_active":"2026-02-25T08:41:18.101098+0000","last_peered":"2026-02-25T08:41:18.101098+0000","last_clean":"2026-02-25T08:41:18.101098+0000","last_became_active":"2026-02-25T08:41:16.125155+0000","last_became_peered":"2026-02-25T08:41:16.125155+0000","last_unstale":"2026-02-25T08:41:18.101098+0000","last_undegraded":"2026-02-25T08:41:18.101098+0000","last_fullsized":"2026-02-25T08:41:18.101098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101259+0000","last_change":"2026-02-25T08:41:18.103989+0000","last_active":"2026-02-25T08:41:18.101259+0000","last_peered":"2026-02-25T08:41:18.101259+0000","last_clean":"2026-02-25T08:41:18.101259+0000","last_became_active":"2026-02-25T08:41:16.133525+0000","last_became_peered":"2026-02-25T08:41:16.133525+0000","last_unstale":"2026-02-25T08:41:18.101259+0000","last_undegraded":"2026-02-25T08:41:18.101259+0000","last_fullsized":"2026-02-25T08:41:18.101259+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":91,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.100204+0000","last_change":"2026-02-25T08:41:09.603704+0000","last_active":"2026-02-25T08:41:18.100204+0000","last_peered":"2026-02-25T08:41:18.100204+0000","last_clean":"2026-02-25T08:41:18.100204+0000","last_became_active":"2026-02-25T08:41:09.591425+0000","last_became_peered":"2026-02-25T08:41:09.591425+0000","last_unstale":"2026-02-25T08:41:18.100204+0000","last_undegraded":"2026-02-25T08:41:18.100204+0000","last_fullsized":"2026-02-25T08:41:18.100204+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:08.549571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:08.549571+0000","last_clean_scrub_stamp":"2026-02-25T08:41:08.549571+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160252022784,"internally_reserved":0,"allocated":3944448,"data_stored":3944448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672972,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":372,"kb_used_data":372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166284,"statfs":{"total":53418655744,"available":53418274816,"internally_reserved":0,"allocated":380928,"data_stored":380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165324,"statfs":{"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1276,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165380,"statfs":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1244,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165412,"statfs":{"total":53418655744,"available":53417381888,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417381888,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:22.211 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:41:22.212 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-25T08:41:22.212 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-25T08:41:22.212 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-25T08:41:22.287 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:22.283+0000 7f1c18b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.289 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:22.287+0000 7f1c18b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.429 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:41:22.429 INFO:teuthology.orchestra.run.trial039.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-25T08:41:22.438 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-25T08:41:22.439 INFO:teuthology.run_tasks:Running task thrashosds... 2026-02-25T08:41:22.456 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-25T08:41:22.457 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}} 2026-02-25T08:41:22.457 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-25T08:41:22.457 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2026-02-25T08:41:22.536 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:22.529+0000 7f652850b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.538 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:22.536+0000 7f652850b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.663 INFO:tasks.thrashosds:Beginning thrashosds... 2026-02-25T08:41:22.663 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:41:22.664 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:22.739 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:22.735+0000 7f3af9db8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.741 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:22.739+0000 7f3af9db8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.855 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:41:22.855 INFO:teuthology.orchestra.run.trial039.stderr:dumped all 2026-02-25T08:41:22.864 INFO:teuthology.orchestra.run.trial039.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-25T08:41:22.377300+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4224,"kb_used_data":4224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662400,"statfs":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.295600"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.100814+0000","last_change":"2026-02-25T08:41:18.104814+0000","last_active":"2026-02-25T08:41:18.100814+0000","last_peered":"2026-02-25T08:41:18.100814+0000","last_clean":"2026-02-25T08:41:18.100814+0000","last_became_active":"2026-02-25T08:41:16.142826+0000","last_became_peered":"2026-02-25T08:41:16.142826+0000","last_unstale":"2026-02-25T08:41:18.100814+0000","last_undegraded":"2026-02-25T08:41:18.100814+0000","last_fullsized":"2026-02-25T08:41:18.100814+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101069+0000","last_change":"2026-02-25T08:41:18.103612+0000","last_active":"2026-02-25T08:41:18.101069+0000","last_peered":"2026-02-25T08:41:18.101069+0000","last_clean":"2026-02-25T08:41:18.101069+0000","last_became_active":"2026-02-25T08:41:16.124173+0000","last_became_peered":"2026-02-25T08:41:16.124173+0000","last_unstale":"2026-02-25T08:41:18.101069+0000","last_undegraded":"2026-02-25T08:41:18.101069+0000","last_fullsized":"2026-02-25T08:41:18.101069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101938+0000","last_change":"2026-02-25T08:41:18.105179+0000","last_active":"2026-02-25T08:41:18.101938+0000","last_peered":"2026-02-25T08:41:18.101938+0000","last_clean":"2026-02-25T08:41:18.101938+0000","last_became_active":"2026-02-25T08:41:16.123155+0000","last_became_peered":"2026-02-25T08:41:16.123155+0000","last_unstale":"2026-02-25T08:41:18.101938+0000","last_undegraded":"2026-02-25T08:41:18.101938+0000","last_fullsized":"2026-02-25T08:41:18.101938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103444+0000","last_change":"2026-02-25T08:41:18.105717+0000","last_active":"2026-02-25T08:41:18.103444+0000","last_peered":"2026-02-25T08:41:18.103444+0000","last_clean":"2026-02-25T08:41:18.103444+0000","last_became_active":"2026-02-25T08:41:16.123587+0000","last_became_peered":"2026-02-25T08:41:16.123587+0000","last_unstale":"2026-02-25T08:41:18.103444+0000","last_undegraded":"2026-02-25T08:41:18.103444+0000","last_fullsized":"2026-02-25T08:41:18.103444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'1","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103630+0000","last_change":"2026-02-25T08:41:18.103630+0000","last_active":"2026-02-25T08:41:18.103630+0000","last_peered":"2026-02-25T08:41:18.103630+0000","last_clean":"2026-02-25T08:41:18.103630+0000","last_became_active":"2026-02-25T08:41:16.124400+0000","last_became_peered":"2026-02-25T08:41:16.124400+0000","last_unstale":"2026-02-25T08:41:18.103630+0000","last_undegraded":"2026-02-25T08:41:18.103630+0000","last_fullsized":"2026-02-25T08:41:18.103630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103342+0000","last_change":"2026-02-25T08:41:18.105717+0000","last_active":"2026-02-25T08:41:18.103342+0000","last_peered":"2026-02-25T08:41:18.103342+0000","last_clean":"2026-02-25T08:41:18.103342+0000","last_became_active":"2026-02-25T08:41:16.127216+0000","last_became_peered":"2026-02-25T08:41:16.127216+0000","last_unstale":"2026-02-25T08:41:18.103342+0000","last_undegraded":"2026-02-25T08:41:18.103342+0000","last_fullsized":"2026-02-25T08:41:18.103342+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101098+0000","last_change":"2026-02-25T08:41:18.103749+0000","last_active":"2026-02-25T08:41:18.101098+0000","last_peered":"2026-02-25T08:41:18.101098+0000","last_clean":"2026-02-25T08:41:18.101098+0000","last_became_active":"2026-02-25T08:41:16.125155+0000","last_became_peered":"2026-02-25T08:41:16.125155+0000","last_unstale":"2026-02-25T08:41:18.101098+0000","last_undegraded":"2026-02-25T08:41:18.101098+0000","last_fullsized":"2026-02-25T08:41:18.101098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101259+0000","last_change":"2026-02-25T08:41:18.103989+0000","last_active":"2026-02-25T08:41:18.101259+0000","last_peered":"2026-02-25T08:41:18.101259+0000","last_clean":"2026-02-25T08:41:18.101259+0000","last_became_active":"2026-02-25T08:41:16.133525+0000","last_became_peered":"2026-02-25T08:41:16.133525+0000","last_unstale":"2026-02-25T08:41:18.101259+0000","last_undegraded":"2026-02-25T08:41:18.101259+0000","last_fullsized":"2026-02-25T08:41:18.101259+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":91,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.100204+0000","last_change":"2026-02-25T08:41:09.603704+0000","last_active":"2026-02-25T08:41:18.100204+0000","last_peered":"2026-02-25T08:41:18.100204+0000","last_clean":"2026-02-25T08:41:18.100204+0000","last_became_active":"2026-02-25T08:41:09.591425+0000","last_became_peered":"2026-02-25T08:41:09.591425+0000","last_unstale":"2026-02-25T08:41:18.100204+0000","last_undegraded":"2026-02-25T08:41:18.100204+0000","last_fullsized":"2026-02-25T08:41:18.100204+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:08.549571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:08.549571+0000","last_clean_scrub_stamp":"2026-02-25T08:41:08.549571+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160252022784,"internally_reserved":0,"allocated":3944448,"data_stored":3944448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":372,"kb_used_data":372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166284,"statfs":{"total":53418655744,"available":53418274816,"internally_reserved":0,"allocated":380928,"data_stored":380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165324,"statfs":{"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1276,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165380,"statfs":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1244,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165412,"statfs":{"total":53418655744,"available":53417381888,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417381888,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:22.865 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:22.944 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:22.939+0000 7fdd75ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.947 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:22.944+0000 7fdd75ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.058 INFO:teuthology.orchestra.run.trial039.stdout: 2026-02-25T08:41:23.058 INFO:teuthology.orchestra.run.trial039.stderr:dumped all 2026-02-25T08:41:23.067 INFO:teuthology.orchestra.run.trial039.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-25T08:41:22.377300+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4224,"kb_used_data":4224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662400,"statfs":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.295600"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.100814+0000","last_change":"2026-02-25T08:41:18.104814+0000","last_active":"2026-02-25T08:41:18.100814+0000","last_peered":"2026-02-25T08:41:18.100814+0000","last_clean":"2026-02-25T08:41:18.100814+0000","last_became_active":"2026-02-25T08:41:16.142826+0000","last_became_peered":"2026-02-25T08:41:16.142826+0000","last_unstale":"2026-02-25T08:41:18.100814+0000","last_undegraded":"2026-02-25T08:41:18.100814+0000","last_fullsized":"2026-02-25T08:41:18.100814+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101069+0000","last_change":"2026-02-25T08:41:18.103612+0000","last_active":"2026-02-25T08:41:18.101069+0000","last_peered":"2026-02-25T08:41:18.101069+0000","last_clean":"2026-02-25T08:41:18.101069+0000","last_became_active":"2026-02-25T08:41:16.124173+0000","last_became_peered":"2026-02-25T08:41:16.124173+0000","last_unstale":"2026-02-25T08:41:18.101069+0000","last_undegraded":"2026-02-25T08:41:18.101069+0000","last_fullsized":"2026-02-25T08:41:18.101069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101938+0000","last_change":"2026-02-25T08:41:18.105179+0000","last_active":"2026-02-25T08:41:18.101938+0000","last_peered":"2026-02-25T08:41:18.101938+0000","last_clean":"2026-02-25T08:41:18.101938+0000","last_became_active":"2026-02-25T08:41:16.123155+0000","last_became_peered":"2026-02-25T08:41:16.123155+0000","last_unstale":"2026-02-25T08:41:18.101938+0000","last_undegraded":"2026-02-25T08:41:18.101938+0000","last_fullsized":"2026-02-25T08:41:18.101938+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103444+0000","last_change":"2026-02-25T08:41:18.105717+0000","last_active":"2026-02-25T08:41:18.103444+0000","last_peered":"2026-02-25T08:41:18.103444+0000","last_clean":"2026-02-25T08:41:18.103444+0000","last_became_active":"2026-02-25T08:41:16.123587+0000","last_became_peered":"2026-02-25T08:41:16.123587+0000","last_unstale":"2026-02-25T08:41:18.103444+0000","last_undegraded":"2026-02-25T08:41:18.103444+0000","last_fullsized":"2026-02-25T08:41:18.103444+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'1","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103630+0000","last_change":"2026-02-25T08:41:18.103630+0000","last_active":"2026-02-25T08:41:18.103630+0000","last_peered":"2026-02-25T08:41:18.103630+0000","last_clean":"2026-02-25T08:41:18.103630+0000","last_became_active":"2026-02-25T08:41:16.124400+0000","last_became_peered":"2026-02-25T08:41:16.124400+0000","last_unstale":"2026-02-25T08:41:18.103630+0000","last_undegraded":"2026-02-25T08:41:18.103630+0000","last_fullsized":"2026-02-25T08:41:18.103630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.103342+0000","last_change":"2026-02-25T08:41:18.105717+0000","last_active":"2026-02-25T08:41:18.103342+0000","last_peered":"2026-02-25T08:41:18.103342+0000","last_clean":"2026-02-25T08:41:18.103342+0000","last_became_active":"2026-02-25T08:41:16.127216+0000","last_became_peered":"2026-02-25T08:41:16.127216+0000","last_unstale":"2026-02-25T08:41:18.103342+0000","last_undegraded":"2026-02-25T08:41:18.103342+0000","last_fullsized":"2026-02-25T08:41:18.103342+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101098+0000","last_change":"2026-02-25T08:41:18.103749+0000","last_active":"2026-02-25T08:41:18.101098+0000","last_peered":"2026-02-25T08:41:18.101098+0000","last_clean":"2026-02-25T08:41:18.101098+0000","last_became_active":"2026-02-25T08:41:16.125155+0000","last_became_peered":"2026-02-25T08:41:16.125155+0000","last_unstale":"2026-02-25T08:41:18.101098+0000","last_undegraded":"2026-02-25T08:41:18.101098+0000","last_fullsized":"2026-02-25T08:41:18.101098+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.101259+0000","last_change":"2026-02-25T08:41:18.103989+0000","last_active":"2026-02-25T08:41:18.101259+0000","last_peered":"2026-02-25T08:41:18.101259+0000","last_clean":"2026-02-25T08:41:18.101259+0000","last_became_active":"2026-02-25T08:41:16.133525+0000","last_became_peered":"2026-02-25T08:41:16.133525+0000","last_unstale":"2026-02-25T08:41:18.101259+0000","last_undegraded":"2026-02-25T08:41:18.101259+0000","last_fullsized":"2026-02-25T08:41:18.101259+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:15.075454+0000","last_clean_scrub_stamp":"2026-02-25T08:41:15.075454+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":91,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-25T08:41:18.100204+0000","last_change":"2026-02-25T08:41:09.603704+0000","last_active":"2026-02-25T08:41:18.100204+0000","last_peered":"2026-02-25T08:41:18.100204+0000","last_clean":"2026-02-25T08:41:18.100204+0000","last_became_active":"2026-02-25T08:41:09.591425+0000","last_became_peered":"2026-02-25T08:41:09.591425+0000","last_unstale":"2026-02-25T08:41:18.100204+0000","last_undegraded":"2026-02-25T08:41:18.100204+0000","last_fullsized":"2026-02-25T08:41:18.100204+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:08.549571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:08.549571+0000","last_clean_scrub_stamp":"2026-02-25T08:41:08.549571+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160255967232,"available":160252022784,"internally_reserved":0,"allocated":3944448,"data_stored":3944448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":372,"kb_used_data":372,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166284,"statfs":{"total":53418655744,"available":53418274816,"internally_reserved":0,"allocated":380928,"data_stored":380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165324,"statfs":{"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1276,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165380,"statfs":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1244,"kb_used_data":1244,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165412,"statfs":{"total":53418655744,"available":53417381888,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417381888,"internally_reserved":0,"allocated":1273856,"data_stored":1273856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:23.068 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:41:23.069 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2026-02-25T08:41:23.145 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:23.141+0000 7f0c1e2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.148 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:23.145+0000 7f0c1e2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.262 INFO:teuthology.orchestra.run.trial039.stdout:epoch 18 2026-02-25T08:41:23.262 INFO:teuthology.orchestra.run.trial039.stdout:fsid 10023e6d-f681-4f84-90aa-0a6f37c6d62c 2026-02-25T08:41:23.262 INFO:teuthology.orchestra.run.trial039.stdout:created 2026-02-25T08:40:58.038353+0000 2026-02-25T08:41:23.262 INFO:teuthology.orchestra.run.trial039.stdout:modified 2026-02-25T08:41:18.088268+0000 2026-02-25T08:41:23.262 INFO:teuthology.orchestra.run.trial039.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:crush_version 5 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:full_ratio 0.95 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:backfillfull_ratio 0.9 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:nearfull_ratio 0.85 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:require_min_compat_client luminous 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:min_compat_client jewel 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:require_osd_release tentacle 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:stretch_mode_enabled false 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:allow_crimson=true 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2026-02-25T08:41:23.263 INFO:teuthology.orchestra.run.trial039.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 18 flags hashpspool,selfmanaged_snaps,crimson stripe_width 0 application rbd read_balance_score 2.00 2026-02-25T08:41:23.264 INFO:teuthology.orchestra.run.trial039.stdout: removed_snaps_queue [2~1] 2026-02-25T08:41:23.264 INFO:teuthology.orchestra.run.trial039.stdout:max_osd 4 2026-02-25T08:41:23.264 INFO:teuthology.orchestra.run.trial039.stdout:osd.0 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.39:6803/1465499150 v2:10.20.193.39:6800/1465499150 exists,up 529301d2-73dd-4079-88ca-4064a8becb6e 2026-02-25T08:41:23.264 INFO:teuthology.orchestra.run.trial039.stdout:osd.1 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.39:6802/148626133 v2:10.20.193.39:6801/148626133 exists,up fcf3ee73-be1b-484d-a5a7-2aee2fa8c451 2026-02-25T08:41:23.264 INFO:teuthology.orchestra.run.trial039.stdout:osd.2 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.91:6805/3625174827 v2:10.20.193.91:6806/3625174827 exists,up 8b58ba7d-53ad-46af-9b4f-bd4f6994d8b0 2026-02-25T08:41:23.264 INFO:teuthology.orchestra.run.trial039.stdout:osd.3 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) v2:10.20.193.91:6803/3250748496 v2:10.20.193.91:6802/3250748496 exists,up f5427ebf-811a-4afc-8c54-a422f25ec213 2026-02-25T08:41:23.273 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.39:6803/1465499150 v2:10.20.193.39:6800/1465499150 exists,up 529301d2-73dd-4079-88ca-4064a8becb6e', 'osd.1 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.39:6802/148626133 v2:10.20.193.39:6801/148626133 exists,up fcf3ee73-be1b-484d-a5a7-2aee2fa8c451', 'osd.2 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.91:6805/3625174827 v2:10.20.193.91:6806/3625174827 exists,up 8b58ba7d-53ad-46af-9b4f-bd4f6994d8b0', 'osd.3 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) v2:10.20.193.91:6803/3250748496 v2:10.20.193.91:6802/3250748496 exists,up f5427ebf-811a-4afc-8c54-a422f25ec213'] 2026-02-25T08:41:23.274 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2026-02-25T08:41:23.349 INFO:teuthology.orchestra.run.trial039.stdout:{ 2026-02-25T08:41:23.349 INFO:teuthology.orchestra.run.trial039.stdout: "name": "mon.a", 2026-02-25T08:41:23.349 INFO:teuthology.orchestra.run.trial039.stdout: "cluster": "ceph", 2026-02-25T08:41:23.349 INFO:teuthology.orchestra.run.trial039.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2026-02-25T08:41:23.349 INFO:teuthology.orchestra.run.trial039.stdout: "admin_socket_mode": "", 2026-02-25T08:41:23.349 INFO:teuthology.orchestra.run.trial039.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2026-02-25T08:41:23.349 INFO:teuthology.orchestra.run.trial039.stdout: "auth_client_required": "cephx, none", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "auth_cluster_required": "cephx", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "auth_debug": "true", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "auth_mon_ticket_ttl": "660.000000", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "auth_service_required": "cephx", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "auth_service_ticket_ttl": "240.000000", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "auth_supported": "cephx", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_aio": "true", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_aio_max_queue_depth": "1024", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_aio_poll_ms": "250", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_aio_reap_max": "16", 2026-02-25T08:41:23.350 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_aio_submit_retry_initial_delay_us": "125", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_aio_submit_retry_max": "16", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_async_discard": "false", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_async_discard_max_pending": "1000000", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_async_discard_threads": "0", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_block_size": "4096", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_debug_aio": "false", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_debug_aio_log_age": "5.000000", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_debug_discard_sleep": "0", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_debug_inflight_ios": "false", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_discard_max_bytes": "10737418240", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_enable_discard": "false", 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_flock_retry": "3", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_flock_retry_interval": "0.100000", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_inject_crash": "0", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_inject_crash_flush_delay": "2", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_ioring": "false", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_ioring_hipri": "false", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_ioring_sqthread_poll": "false", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_max_discard_length": "2147483648", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_nvme_unbind_from_kernel": "false", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_read_buffer_alignment": "4096", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_read_preallocated_huge_buffers": "", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2026-02-25T08:41:23.352 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_stalled_read_warn_threshold": "1", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bdev_type": "", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_alloc_size": "1048576", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_allocator": "hybrid", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_buffered_io": "true", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_check_for_zeros": "false", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_check_volume_selector_often": "false", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_check_volume_selector_on_umount": "false", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_compact_log_sync": "false", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_log_compact_min_size": "16777216", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_log_replay_check_allocations": "true", 2026-02-25T08:41:23.353 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_max_log_runway": "4194304", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_max_prefetch": "1048576", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_min_flush_size": "524288", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_min_log_runway": "1048576", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_replay_recovery": "false", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_replay_recovery_disable_compact": "false", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_shared_alloc_size": "65536", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_sync_write": "false", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluefs_wal_envelope_mode": "true", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_allocation_from_file": "true", 2026-02-25T08:41:23.354 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_allocator": "hybrid", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_allocator_lookup_policy": "auto", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_async_db_compaction": "true", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_bdev_label_multi": "true", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_bdev_label_require_all": "true", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_bitmapallocator_span_size": "1024", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_blobid_prealloc": "10240", 2026-02-25T08:41:23.355 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_block_create": "true", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_block_db_create": "false", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_block_db_path": "", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_block_db_size": "0", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_block_path": "", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_block_preallocate_file": "false", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_block_size": "107374182400", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_block_wal_create": "false", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_block_wal_path": "", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_block_wal_size": "100663296", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_bluefs": "true", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_bluefs_env_mirror": "false", 2026-02-25T08:41:23.356 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_bluefs_max_free": "10737418240", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_autotune": "true", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_autotune_interval": "5.000000", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_kv_ratio": "0.450000", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_meta_ratio": "0.450000", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_size": "0", 2026-02-25T08:41:23.357 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_size_hdd": "1073741824", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_size_ssd": "3221225472", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_trim_interval": "0.050000", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cache_type": "2q", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_clone_cow": "true", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_compression_algorithm": "snappy", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_compression_max_blob_size": "0", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_compression_min_blob_size": "0", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2026-02-25T08:41:23.358 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_compression_mode": "none", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_compression_required_ratio": "0.875000", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_csum_type": "crc32c", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_enforce_settings": "default", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_extent_map_encode_check": "false", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_freelist": "false", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_fsck_abort": "false", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_inject_read_err": "false", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_legacy_omap": "false", 2026-02-25T08:41:23.359 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_no_reuse_blocks": "false", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_omit_block_device_write": "false", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_omit_kv_commit": "false", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_onode_segmentation_random": "false", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_prefragment_max": "1048576", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_random_read_err": "0.000000", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_small_allocations": "0", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_default_buffered_read": "true", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_default_buffered_write": "false", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_deferred_batch_ops": "0", 2026-02-25T08:41:23.360 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_discard_on_mkfs": "true", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_elastic_shared_blobs": "true", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_extent_map_shard_max_size": "1200", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_extent_map_shard_min_size": "150", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_extent_map_shard_target_size": "500", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fail_eio": "false", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fragmentation_check_period": "3600", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_freelist_blocks_per_key": "128", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2026-02-25T08:41:23.361 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_on_mkfs": "true", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_on_mount": "false", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_on_mount_deep": "false", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_on_umount": "false", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_on_umount_deep": "false", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_quick_fix_threads": "2", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_gc_enable_blob_threshold": "0", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_gc_enable_total_threshold": "0", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2026-02-25T08:41:23.362 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_ignore_data_csum": "false", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_kvbackend": "rocksdb", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_log_collection_list_age": "60.000000", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_log_op_age": "5.000000", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_max_alloc_size": "0", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_max_blob_size": "0", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_max_blob_size_hdd": "65536", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_max_blob_size_ssd": "65536", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_max_defer_interval": "3.000000", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_max_deferred_txc": "32", 2026-02-25T08:41:23.363 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_min_alloc_size": "0", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_min_alloc_size_hdd": "4096", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_min_alloc_size_ssd": "4096", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_nid_prealloc": "1024", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_onode_segment_size": "0", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_prefer_deferred_size": "0", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_qfsck_on_mount": "true", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_recompression_min_gain": "1.200000", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_retry_disk_reads": "3", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_rocksdb_cf": "true", 2026-02-25T08:41:23.364 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_rocksdb_options_annex": "", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_slow_ops_warn_threshold": "1", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_spdk_coremask": "0x1", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_spdk_io_sleep": "5", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_spdk_max_io_completion": "0", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_spdk_mem": "512", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_sync_submit_transaction": "false", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_throttle_bytes": "67108864", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_throttle_cost_per_io": "0", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2026-02-25T08:41:23.365 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_throttle_trace_rate": "0.000000", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_tracing": "false", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_volume_selection_reserved": "0", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_warn_on_free_fragmentation": "0.800000", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_warn_on_legacy_statfs": "true", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2026-02-25T08:41:23.366 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_write_v2": "false", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_write_v2_random": "false", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "bluestore_zero_block_detection": "false", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "breakpad": "false", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "ceph_assert_supresssions": "", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "cephadm_path": "/usr/sbin/cephadm", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "cephfs_mirror_action_update_interval": "2", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "cephfs_mirror_directory_scan_interval": "10", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "cephfs_mirror_mount_timeout": "10", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "cephfs_mirror_perf_stats_prio": "5", 2026-02-25T08:41:23.367 INFO:teuthology.orchestra.run.trial039.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephsqlite_blocklist_dead_locker": "true", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephsqlite_lock_renewal_interval": "2000", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephx_cluster_require_signatures": "false", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephx_cluster_require_version": "2", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephx_require_signatures": "false", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephx_require_version": "2", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephx_service_require_signatures": "false", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephx_service_require_version": "2", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "cephx_sign_messages": "true", 2026-02-25T08:41:23.368 INFO:teuthology.orchestra.run.trial039.stdout: "chdir": "", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_acl_type": "", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_asio_thread_count": "2", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_cache_mid": "0.750000", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_cache_size": "16384", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_caps_release_delay": "5", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_check_pool_perm": "true", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_collect_and_send_global_metrics": "false", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_debug_force_sync_read": "false", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_debug_getattr_caps": "false", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_debug_inject_features": "", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_debug_inject_tick_delay": "0", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_die_on_failed_dentry_invalidate": "true", 2026-02-25T08:41:23.369 INFO:teuthology.orchestra.run.trial039.stdout: "client_die_on_failed_remount": "false", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_dirsize_rbytes": "true", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_force_lazyio": "false", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_fs": "", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_fscrypt_as": "true", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_fscrypt_dummy_encryption": "false", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_inject_fixed_oldest_tid": "false", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_inject_release_failure": "false", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_inject_write_delay_secs": "0", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_max_inline_size": "4096", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_max_retries_on_remount_failure": "5", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_mds_namespace": "", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_metadata": "", 2026-02-25T08:41:23.370 INFO:teuthology.orchestra.run.trial039.stdout: "client_mount_gid": "-1", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_mount_timeout": "300", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_mount_uid": "-1", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_mountpoint": "/", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_notify_timeout": "10", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_oc": "true", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_oc_max_dirty": "104857600", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_oc_max_dirty_age": "5.000000", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_oc_max_objects": "1000", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_oc_size": "209715200", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_oc_target_dirty": "8388608", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_permissions": "true", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_quota": "true", 2026-02-25T08:41:23.371 INFO:teuthology.orchestra.run.trial039.stdout: "client_quota_df": "true", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_readahead_max_bytes": "0", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_readahead_max_periods": "4", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_readahead_min": "131072", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_reconnect_stale": "false", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_respect_subvolume_snapshot_visibility": "false", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_shutdown_timeout": "30", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_snapdir": ".snap", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_tick_interval": "1", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_trace": "", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_try_dentry_invalidate": "false", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_use_faked_inos": "false", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "client_use_random_mds": "false", 2026-02-25T08:41:23.372 INFO:teuthology.orchestra.run.trial039.stdout: "clog_to_graylog": "false", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "clog_to_graylog_host": "127.0.0.1", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "clog_to_graylog_port": "12201", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "clog_to_monitors": "default=true", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "clog_to_syslog": "false", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "clog_to_syslog_level": "info", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "cluster_addr": "-", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "cluster_network": "", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "cluster_network_interface": "", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "compressor_zlib_isal": "false", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "compressor_zlib_level": "5", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "compressor_zlib_winsize": "-15", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "compressor_zstd_level": "1", 2026-02-25T08:41:23.373 INFO:teuthology.orchestra.run.trial039.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crash_dir": "/var/lib/ceph/crash", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_allow_pg_split": "true", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_bluestore_cpu_set": "", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_bluestore_num_threads": "6", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_cpu_num": "0", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_cpu_set": "", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_osd_obc_lru_size": "512", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_osd_scheduler_concurrency": "0", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_osd_stat_interval": "0", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_poll_mode": "false", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_reactor_idle_poll_time_us": "200", 2026-02-25T08:41:23.374 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_reactor_io_latency_goal_ms": "0.000000", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "crimson_reactor_task_quota_ms": "0.500000", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "crush_location": "", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "crush_location_hook": "", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "crush_location_hook_timeout": "10", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "d4n_writecache_enabled": "false", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "daemonize": "false", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "daos_pool": "tank", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "dbstore_db_name_prefix": "dbstore", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "debug_alienstore": "0/5", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "debug_allow_any_pool_priority": "false", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "debug_asok": "1/5", 2026-02-25T08:41:23.375 INFO:teuthology.orchestra.run.trial039.stdout: "debug_asok_assert_abort": "false", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_asserts_on_shutdown": "true", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_auth": "20/20", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_bdev": "1/3", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_bluefs": "1/5", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_bluestore": "1/5", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_bluestore_compression": "1/5", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_buffer": "0/1", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_ceph_dedup": "0/5", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_ceph_exporter": "1/5", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_cephfs_mirror": "0/5", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_cephsqlite": "0/5", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_client": "0/5", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_compressor": "1/5", 2026-02-25T08:41:23.376 INFO:teuthology.orchestra.run.trial039.stdout: "debug_context": "0/1", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_crimson_interrupt": "0/5", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_crush": "1/1", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_crypto": "1/5", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_cyanstore": "0/5", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_deliberately_leak_memory": "false", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_disable_randomized_ping": "false", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_dpdk": "1/5", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_eventtrace": "1/5", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_filer": "0/1", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_filestore": "1/3", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_finisher": "1/1", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_fuse": "1/5", 2026-02-25T08:41:23.377 INFO:teuthology.orchestra.run.trial039.stdout: "debug_heartbeat_testing_span": "0", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_heartbeatmap": "1/5", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_immutable_obj_cache": "0/5", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_javaclient": "1/5", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_journal": "1/3", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_journaler": "0/5", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_kstore": "1/5", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_lockdep": "0/1", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mclock": "1/5", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mds": "1/5", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mds_balancer": "1/5", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mds_locker": "1/5", 2026-02-25T08:41:23.378 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mds_log": "1/5", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mds_log_expire": "1/5", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mds_migrator": "1/5", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mds_quiesce": "3/5", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_memstore": "1/5", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mgr": "2/5", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mgrc": "1/5", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_mon": "20/20", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_monc": "0/10", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_ms": "1/1", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_none": "0/5", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_objclass": "0/5", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_objectcacher": "0/5", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_objecter": "0/1", 2026-02-25T08:41:23.379 INFO:teuthology.orchestra.run.trial039.stdout: "debug_optracker": "0/5", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_osd": "1/5", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_paxos": "20/20", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_perfcounter": "1/5", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_prioritycache": "1/5", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rados": "0/5", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rbd": "0/5", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rbd_mirror": "0/5", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rbd_pwl": "0/5", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rbd_replay": "0/5", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_refs": "0/0", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_reserver": "1/1", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw": "1/5", 2026-02-25T08:41:23.380 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw_access": "1/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw_bucket_logging": "1/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw_datacache": "1/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw_dbstore": "1/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw_dedup": "1/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw_flight": "1/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw_lifecycle": "1/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw_notification": "1/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw_restore": "1/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rgw_sync": "1/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_rocksdb": "4/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore": "0/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_backref": "0/5", 2026-02-25T08:41:23.381 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_cache": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_cleaner": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_device": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_epm": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_fixedkv_tree": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_journal": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_lba": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_odata": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_omap": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_onode": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_t": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_seastore_tm": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_striper": "0/1", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_test": "0/5", 2026-02-25T08:41:23.382 INFO:teuthology.orchestra.run.trial039.stdout: "debug_throttle": "1/1", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "debug_timer": "0/1", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "debug_tp": "0/5", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "debug_trace": "1/5", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "defer_client_eviction_on_laggy_osds": "false", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "device_failure_prediction_mode": "none", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "ec_extent_cache_size": "10485760", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "ec_pdw_write_mode": "0", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "enable_availability_tracking": "false", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "err_to_graylog": "false", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "err_to_journald": "false", 2026-02-25T08:41:23.383 INFO:teuthology.orchestra.run.trial039.stdout: "err_to_stderr": "true", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "err_to_syslog": "false", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "event_tracing": "false", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "exporter_addr": "0.0.0.0", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "exporter_cert_file": "", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "exporter_http_port": "9926", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "exporter_key_file": "", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "exporter_prio_limit": "5", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "exporter_sock_dir": "/var/run/ceph/", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "exporter_sort_metrics": "true", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "exporter_stats_period": "5", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "fake_statfs_for_testing": "0", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "fatal_signal_handlers": "true", 2026-02-25T08:41:23.384 INFO:teuthology.orchestra.run.trial039.stdout: "filer_max_purge_ops": "10", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filer_max_truncate_ops": "128", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_apply_finisher_threads": "1", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_blackhole": "false", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_btrfs_clone_range": "true", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_btrfs_snap": "true", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_caller_concurrency": "10", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_collect_device_partition_information": "true", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_commit_timeout": "600.000000", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_debug_inject_read_err": "false", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_debug_omap_check": "false", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_debug_verify_split": "false", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_dump_file": "", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_expected_throughput_ops": "200.000000", 2026-02-25T08:41:23.385 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_fadvise": "true", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_fail_eio": "true", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_fd_cache_shards": "16", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_fd_cache_size": "128", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_fiemap": "false", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_fiemap_threshold": "4096", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_fsync_flushes_journal_data": "false", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_index_retry_probability": "0.000000", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_inject_stall": "0", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_journal_parallel": "false", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_journal_trailing": "false", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_journal_writeahead": "false", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_kill_at": "0", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_alloc_hint_size": "1048576", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_inline_xattr_size": "0", 2026-02-25T08:41:23.386 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_inline_xattr_size_other": "512", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_inline_xattrs": "0", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_inline_xattrs_other": "2", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_inline_xattrs_xfs": "10", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_sync_interval": "5.000000", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_xattr_value_size": "0", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_xattr_value_size_other": "1024", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_merge_threshold": "-10", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_min_sync_interval": "0.010000", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_odsync_write": "false", 2026-02-25T08:41:23.387 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_omap_backend": "rocksdb", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_omap_backend_path": "", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_omap_header_cache_size": "1024", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_ondisk_finisher_threads": "1", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_op_thread_suicide_timeout": "180", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_op_thread_timeout": "60", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_op_threads": "2", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_punch_hole": "false", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_queue_high_threshhold": "0.900000", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_queue_low_threshhold": "0.300000", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_queue_max_bytes": "104857600", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2026-02-25T08:41:23.388 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_queue_max_ops": "50", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_seek_data_hole": "false", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_sloppy_crc": "false", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_sloppy_crc_block_size": "65536", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_splice": "false", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_split_multiple": "2", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_split_rand_factor": "20", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_update_to": "1000", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2026-02-25T08:41:23.389 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_enable": "true", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_xfs_extsize": "false", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "filestore_zfs_snap": "false", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "fio_dir": "/tmp/fio", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "fsid": "10023e6d-f681-4f84-90aa-0a6f37c6d62c", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_allow_other": "true", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_atomic_o_trunc": "true", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_big_writes": "true", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_debug": "false", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_default_permissions": "false", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_disable_pagecache": "false", 2026-02-25T08:41:23.390 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_max_write": "0", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_multithreaded": "true", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_require_active_mds": "true", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_set_user_groups": "true", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_splice_move": "true", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_splice_read": "true", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_splice_write": "true", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_syncfs_on_mksnap": "true", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "fuse_use_invalidate_cb": "true", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "gss_target_name": "ceph", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "heartbeat_file": "", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "heartbeat_inject_failure": "0", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "heartbeat_interval": "5", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "host": "trial039", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_max_inflight_ops": "128", 2026-02-25T08:41:23.391 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_max_size": "1073741824", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_qos_bps_burst": "0", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_qos_bps_limit": "0", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_qos_iops_burst": "0", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_qos_iops_limit": "0", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "immutable_object_cache_watermark": "0.900000", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "inject_early_sigterm": "false", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "jaeger_agent_port": "6799", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "jaeger_tracing_enable": "false", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "journal_aio": "true", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "journal_align_min_size": "65536", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "journal_block_align": "true", 2026-02-25T08:41:23.392 INFO:teuthology.orchestra.run.trial039.stdout: "journal_block_size": "4096", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_dio": "true", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_discard": "false", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_force_aio": "false", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_ignore_corruption": "false", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_max_write_bytes": "10485760", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_max_write_entries": "100", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_replay_from": "0", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_throttle_high_multiple": "0.000000", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_throttle_high_threshhold": "0.900000", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_throttle_low_threshhold": "0.600000", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_throttle_max_multiple": "0.000000", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_write_header_frequency": "0", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journal_zero_on_create": "false", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journaler_prefetch_periods": "10", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journaler_prezero_periods": "5", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "journaler_write_head_interval": "15", 2026-02-25T08:41:23.393 INFO:teuthology.orchestra.run.trial039.stdout: "key": "", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "keyfile": "", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "librados_thread_count": "2", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "lockdep": "false", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "lockdep_force_backtrace": "false", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_coarse_timestamps": "true", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_flush_on_exit": "false", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_graylog_host": "127.0.0.1", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_graylog_port": "12201", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_max_new": "1000", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_max_recent": "10000", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_stderr_prefix": "", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_stop_at_utilization": "0.970000", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_to_file": "true", 2026-02-25T08:41:23.394 INFO:teuthology.orchestra.run.trial039.stdout: "log_to_graylog": "false", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "log_to_journald": "false", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "log_to_stderr": "false", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "log_to_syslog": "false", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "max_rotating_auth_attempts": "10", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_action_on_write_error": "1", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_allow_async_dirops": "true", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_allow_batched_ops": "true", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_alternate_name_max": "8192", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_asio_thread_count": "2", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_export_pin": "true", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_fragment_dirs": "true", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_fragment_interval": "5", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_fragment_size_max": "100000", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_idle_threshold": "0.000000", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_interval": "10", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_max": "-1", 2026-02-25T08:41:23.395 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_max_until": "-1", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_merge_size": "50", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_midchunk": "0.300000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_min_rebalance": "0.100000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_min_start": "0.200000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_minchunk": "0.001000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_mode": "0", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_need_max": "1.200000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_need_min": "0.800000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_overload_epochs": "2", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_replicate_threshold": "8000.000000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_sample_interval": "3.000000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_split_bits": "3", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_split_rd": "25000.000000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_split_size": "10000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_split_wr": "10000.000000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_target_decay": "10.000000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_beacon_grace": "15.000000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_beacon_interval": "4.000000", 2026-02-25T08:41:23.396 INFO:teuthology.orchestra.run.trial039.stdout: "mds_beacon_mon_down_grace": "60", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_memory_limit": "4294967296", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_mid": "0.700000", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_quiesce_delay": "0", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_quiesce_sleep": "200", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_quiesce_splitauth": "true", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_quiesce_threshold": "524288", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_release_free_interval": "10", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_reservation": "0.050000", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_trim_decay_rate": "1.000000", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_trim_interval": "1", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cache_trim_threshold": "262144", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_client_delegate_inos_pct": "50", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_client_prealloc_inos": "1000", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_connect_bootstrapping": "false", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_damage_table_max_entries": "10000", 2026-02-25T08:41:23.397 INFO:teuthology.orchestra.run.trial039.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_debug_auth_pins": "false", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_debug_frag": "false", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_debug_scatterstat": "false", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_debug_subtrees": "false", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_decay_halflife": "5.000000", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_default_dir_hash": "2", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_defer_session_stale": "true", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_delay_journal_replay_for_testing": "0", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_deny_all_reconnect": "false", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_dir_keys_per_op": "16384", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_dir_max_commit_size": "10", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_dir_max_entries": "0", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_dir_prefetch": "true", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_dirstat_min_interval": "1.000000", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_dump_cache_after_rejoin": "false", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_dump_cache_on_map": "false", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_dump_cache_threshold_file": "0", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_early_reply": "true", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_enable_op_tracker": "true", 2026-02-25T08:41:23.398 INFO:teuthology.orchestra.run.trial039.stdout: "mds_enforce_unique_name": "true", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_export_ephemeral_distributed": "true", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_export_ephemeral_random": "true", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_export_ephemeral_random_max": "0.010000", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_extraordinary_events_dump_interval": "0", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_forward_all_requests_to_auth": "false", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_freeze_tree_timeout": "30.000000", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_fscrypt_last_block_max_size": "4096", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_go_bad_corrupt_dentry": "true", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_health_cache_threshold": "1.500000", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_health_summarize_threshold": "10", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_heartbeat_grace": "15.000000", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_heartbeat_reset_grace": "1000", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_inject_health_dummy": "false", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_inject_migrator_session_race": "false", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_inject_skip_replaying_inotable": "false", 2026-02-25T08:41:23.399 INFO:teuthology.orchestra.run.trial039.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_join_fs": "", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_journal_format": "1", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_after_journal_logs_flushed": "false", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_create_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_dirfrag_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_export_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_import_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_journal_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_journal_expire_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_journal_replay_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_link_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_mdstable_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_openc_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_rename_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_kill_shutdown_at": "0", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_event_large_threshold": "524288", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_events_per_segment": "1024", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_max_events": "-1", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_max_segments": "128", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_minor_segments_per_major_segment": "16", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_pause": "false", 2026-02-25T08:41:23.400 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_segment_size": "0", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_skip_corrupt_events": "false", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_skip_unbounded_events": "false", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_trim_decay_rate": "1.000000", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_trim_threshold": "128", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_trim_upkeep_interval": "1000", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_log_warn_factor": "2.000000", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_max_caps_per_client": "1048576", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_max_completed_flushes": "100000", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_max_completed_requests": "100000", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_max_export_size": "20971520", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_max_file_recover": "32", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_max_purge_files": "64", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_max_purge_ops": "8192", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_max_scrub_ops_in_progress": "5", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_max_snaps_per_dir": "100", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_metrics_update_interval": "2", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_min_caps_per_client": "100", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_min_caps_working_set": "10000", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_mon_shutdown_timeout": "5.000000", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_numa_node": "-1", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_oft_prefetch_dirfrags": "false", 2026-02-25T08:41:23.401 INFO:teuthology.orchestra.run.trial039.stdout: "mds_op_complaint_time": "30.000000", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_op_history_duration": "600", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_op_history_size": "20", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_op_history_slow_op_size": "20", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_op_history_slow_op_threshold": "10", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_op_log_threshold": "5", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_ping_grace": "15", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_ping_interval": "5", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_recall_global_max_decay_threshold": "131072", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_recall_max_caps": "30000", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_recall_max_decay_rate": "1.500000", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_recall_max_decay_threshold": "131072", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_recall_warning_decay_rate": "60.000000", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_recall_warning_threshold": "262144", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_reconnect_timeout": "45.000000", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_replay_interval": "1.000000", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_replay_unsafe_with_closed_session": "false", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_request_load_average_decay_rate": "60.000000", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_root_ino_gid": "0", 2026-02-25T08:41:23.402 INFO:teuthology.orchestra.run.trial039.stdout: "mds_root_ino_uid": "0", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_scatter_nudge_interval": "5.000000", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_scrub_stats_review_period": "1", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_server_dispatch_client_request_delay": "0", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_session_blocklist_on_evict": "true", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_session_blocklist_on_timeout": "true", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_session_cache_liveness_magnitude": "10", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_session_cap_acquisition_throttle": "100000", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_session_metadata_threshold": "16777216", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_sessionmap_keys_per_op": "1024", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_shutdown_check": "0", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_skip_ino": "0", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_sleep_rank_change": "0.000000", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_snap_max_uid": "4294967294", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_snap_min_uid": "0", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_snap_rstat": "false", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_standby_replay_damaged": "false", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_symlink_recovery": "true", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_task_status_update_interval": "2.000000", 2026-02-25T08:41:23.403 INFO:teuthology.orchestra.run.trial039.stdout: "mds_thrash_exports": "0", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mds_thrash_fragments": "0", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mds_tick_interval": "5.000000", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mds_use_global_snaprealm_seq_for_subvol": "true", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mds_valgrind_exit": "false", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mds_verify_backtrace": "1", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mds_verify_scatter": "false", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mds_wipe_ino_prealloc": "false", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mds_wipe_sessions": "false", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mempool_debug": "false", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "memstore_debug_omit_block_device_write": "false", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "memstore_device_bytes": "1073741824", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "memstore_page_set": "false", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "memstore_page_size": "65536", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_client_bytes": "134217728", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_client_messages": "512", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_connect_retry_interval": "1.000000", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_disabled_modules": "", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_enable_op_tracker": "true", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_initial_modules": "iostat nfs", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_max_pg_creating": "1024", 2026-02-25T08:41:23.404 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_max_pg_num_change": "128", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_mds_bytes": "134217728", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_mds_messages": "128", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_mon_bytes": "134217728", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_mon_messages": "128", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_num_op_tracker_shard": "32", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_op_complaint_time": "30.000000", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_op_history_duration": "600", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_op_history_size": "20", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_op_history_slow_op_size": "20", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_op_log_threshold": "5", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_osd_bytes": "536870912", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_osd_messages": "8192", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_pool": "true", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_service_beacon_grace": "60.000000", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_standby_modules": "true", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_stats_period": "5", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_stats_threshold": "5", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_tick_period": "2", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mgr_ttl_cache_expire_seconds": "0", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mon_accept_timeout_factor": "2.000000", 2026-02-25T08:41:23.405 INFO:teuthology.orchestra.run.trial039.stdout: "mon_allow_pool_delete": "true", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_allow_pool_size_one": "true", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_auth_validate_all_caps": "true", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_bytes": "104857600", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_directed_command_retry": "2", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_hunt_interval": "3.000000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_hunt_on_resend": "true", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_hunt_parallel": "3", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_log_interval": "1.000000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_max_log_entries_per_message": "1000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_ping_interval": "10.000000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_ping_timeout": "30.000000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_client_target_rank": "-1", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_clock_drift_allowed": "1.000000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cluster_log_level": "debug", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cluster_log_to_file": "true", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cluster_log_to_graylog": "false", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2026-02-25T08:41:23.406 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cluster_log_to_graylog_port": "12201", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cluster_log_to_journald": "false", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cluster_log_to_stderr": "false", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cluster_log_to_syslog": "default=false", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_compact_on_bootstrap": "false", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_compact_on_start": "false", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_compact_on_trim": "true", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_con_tracker_persist_interval": "10", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_con_tracker_score_halflife": "43200", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_config_key_max_entry_size": "65536", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_cpu_threads": "4", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_crush_min_required_version": "hammer", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_daemon_bytes": "419430400", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_data_avail_crit": "5", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_data_avail_warn": "5", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_data_size_warn": "16106127360", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_block_osdmap_trim": "false", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_deprecated_as_obsolete": "false", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_dump_json": "false", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_dump_transactions": "false", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_extra_checks": "false", 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_no_initial_persistent_features": "false", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_no_require_squid": "false", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_no_require_tentacle": "false", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_delta_reset_interval": "10.000000", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_dns_srv_name": "ceph-mon", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_down_added_grace": "180", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_down_mkfs_grace": "120", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_down_uptime_grace": "60", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_election_default_strategy": "1", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_election_timeout": "5.000000", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_elector_ping_divisor": "2", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_elector_ping_timeout": "2.000000", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_enable_op_tracker": "true", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_fake_pool_delete": "false", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_force_quorum_join": "false", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_fsmap_prune_threshold": "300", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_globalid_prealloc": "10000", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_health_detail_to_clog": "false", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_health_log_update_period": "5", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_health_max_detail": "50", 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial039.stdout: "mon_health_to_clog": "true", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_health_to_clog_interval": "600", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_host": "10.20.193.39", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_host_override": "", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_initial_members": "", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_inject_transaction_delay_max": "10.000000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_keyvaluedb": "rocksdb", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_lease": "5.000000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_lease_renew_interval_factor": "0.600000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_log_full_interval": "50", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_log_max": "10000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_log_max_summary": "50", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_max_log_entries_per_event": "4096", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_max_log_epochs": "500", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_max_mdsmap_epochs": "500", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_max_mgrmap_epochs": "500", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_max_nvmeof_epochs": "500", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_max_osd": "10000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_max_pg_per_osd": "10000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_max_pool_pg_num": "65536", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_max_snap_prune_per_epoch": "100", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_mds_blocklist_interval": "86400.000000", 2026-02-25T08:41:23.409 INFO:teuthology.orchestra.run.trial039.stdout: "mon_mds_force_trim_to": "0", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_mds_skip_sanity": "false", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_memory_autotune": "true", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_memory_target": "2147483648", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_mgr_beacon_grace": "30", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_mgr_digest_period": "5", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_mgr_inactive_grace": "60", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_mgr_mkfs_grace": "240", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_min_osdmap_epochs": "50", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_netsplit_grace_period": "9", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_nvmeofgw_beacon_grace": "7", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_nvmeofgw_beacons_till_ack": "15", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_nvmeofgw_delete_grace": "900", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_nvmeofgw_skip_failovers_interval": "16", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_nvmeofgw_wrong_map_ignore_sec": "15", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_op_complaint_time": "30", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_op_history_duration": "600", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_op_history_size": "20", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_op_history_slow_op_size": "20", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_op_history_slow_op_threshold": "10", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_op_log_threshold": "5", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_adjust_down_out_interval": "true", 2026-02-25T08:41:23.410 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_auto_mark_in": "false", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_auto_mark_new_in": "true", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_cache_size": "500", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_cache_size_min": "134217728", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_crush_smoke_test": "true", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_destroyed_out_interval": "600", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_down_out_interval": "600", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_down_out_subtree_limit": "rack", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_force_trim_to": "0", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_full_ratio": "0.950000", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_laggy_halflife": "3600", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_laggy_max_interval": "300", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_laggy_weight": "0.300000", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_max_initial_pgs": "1024", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_min_down_reporters": "2", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_min_in_ratio": "0.750000", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_min_up_ratio": "0.300000", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_nearfull_ratio": "0.850000", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_prime_pg_temp": "true", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_report_timeout": "900", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_reporter_subtree_level": "osd", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_warn_num_repaired": "10", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osd_warn_op_age": "32.000000", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osdmap_full_prune_enabled": "true", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osdmap_full_prune_interval": "2", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osdmap_full_prune_min": "15", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_osdmap_full_prune_txsize": "2", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_pg_stuck_threshold": "60", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_pg_warn_min_objects": "10000", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_pg_warn_min_per_osd": "0", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_pg_warn_min_pool_objects": "1000", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_pool_quota_crit_threshold": "0", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_pool_quota_warn_threshold": "0", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_probe_timeout": "2.000000", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_reweight_max_change": "0.050000", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_reweight_max_osds": "4", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_reweight_min_bytes_per_osd": "10", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_reweight_min_pgs_per_osd": "4", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_scrub_interval": "86400", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_scrub_max_keys": "100", 2026-02-25T08:41:23.412 INFO:teuthology.orchestra.run.trial039.stdout: "mon_scrub_timeout": "300", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_session_timeout": "300", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_smart_report_timeout": "5", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_stat_smooth_intervals": "6", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_stretch_pool_min_size": "2", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_stretch_pool_size": "4", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_subscribe_interval": "86400.000000", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_sync_debug": "false", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_sync_max_payload_keys": "2000", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_sync_max_payload_size": "1048576", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_sync_provider_kill_at": "0", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_sync_requester_kill_at": "0", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_sync_timeout": "60.000000", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_target_pg_per_osd": "100", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_tick_interval": "5", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_timecheck_interval": "300.000000", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_timecheck_skew_interval": "30.000000", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_use_min_delay_socket": "false", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_older_version_delay": "604800", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_filestore_osds": "false", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2026-02-25T08:41:23.413 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_misplaced": "false", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_older_version": "true", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_pool_no_app": "true", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_pool_no_app_grace": "300", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_pool_no_redundancy": "false", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_on_too_few_osds": "false", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "monmap": "", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "motr_tracing_enabled": "false", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_op_threads": "3", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_buffer_size": "131072", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_cm": "false", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_device_name": "", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_dscp": "96", 2026-02-25T08:41:23.414 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_enable_hugepage": "false", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_gid_idx": "0", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_local_gid": "", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_polling_us": "1000", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_port_num": "1", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_receive_buffers": "32768", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_receive_queue_len": "4096", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_roce_ver": "1", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_send_buffers": "1024", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_sl": "3", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_support_srq": "true", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_rdma_type": "ib", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_async_reap_threshold": "5", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_bind_before_connect": "false", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_bind_ipv4": "true", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_bind_ipv6": "false", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_bind_msgr1": "true", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_bind_msgr2": "true", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_bind_port_max": "7568", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_bind_port_min": "6800", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_bind_prefer_ipv4": "false", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_bind_retry_count": "3", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_bind_retry_delay": "5", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_blackhole_client": "false", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_blackhole_mds": "false", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_blackhole_mgr": "false", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_blackhole_mon": "false", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_blackhole_osd": "false", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_client_mode": "crc", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_client_throttle_retry_time_interval": "5000", 2026-02-25T08:41:23.415 INFO:teuthology.orchestra.run.trial039.stdout: "ms_cluster_mode": "crc", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_cluster_type": "", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_compress_secure": "false", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_connection_idle_timeout": "900", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_connection_ready_timeout": "10", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_crc_data": "true", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_crc_header": "true", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_die_on_bad_msg": "false", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_die_on_bug": "true", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_die_on_old_message": "true", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_die_on_skipped_message": "false", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_die_on_unhandled_msg": "false", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dispatch_throttle_bytes": "104857600", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_coremask": "0xF", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_debug_allow_loopback": "false", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_devs_allowlist": "", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_enable_tso": "true", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_host_ipv4_addr": "", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_hugepages": "", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_hw_flow_control": "true", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_lro": "true", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_memory_channel": "4", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_pmd": "", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_port_id": "0", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dump_corrupt_message_level": "1", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_dump_on_send": "false", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_initial_backoff": "0.200000", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_inject_delay_max": "1.000000", 2026-02-25T08:41:23.416 INFO:teuthology.orchestra.run.trial039.stdout: "ms_inject_delay_probability": "0.000000", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_inject_delay_type": "", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_inject_internal_delays": "0.000000", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_inject_network_congestion": "0", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_inject_socket_failures": "0", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_learn_addr_from_peer": "true", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_max_accept_failures": "4", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_max_backoff": "15.000000", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_mon_client_mode": "crc", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_mon_cluster_mode": "crc", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_mon_service_mode": "crc", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_osd_compress_min_size": "1024", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_osd_compress_mode": "none", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_osd_compression_algorithm": "snappy", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_pq_min_cost": "65536", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_public_type": "", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_service_mode": "crc", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_tcp_listen_backlog": "512", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_tcp_nodelay": "true", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_tcp_prefetch_max_size": "65536", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_tcp_rcvbuf": "0", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_time_events_min_wait_interval": "1000", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "ms_type": "async+posix", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "no_config_file": "false", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "nvmeof_mon_client_connect_panic": "30", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "nvmeof_mon_client_tick_period": "1", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "objecter_completion_locks_per_session": "32", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "objecter_debug_inject_relock_delay": "false", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "objecter_inflight_op_bytes": "104857600", 2026-02-25T08:41:23.417 INFO:teuthology.orchestra.run.trial039.stdout: "objecter_inflight_ops": "1024", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "objecter_inject_no_watch_ping": "false", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "objecter_retry_writes_after_first_reply": "false", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "objecter_tick_interval": "5.000000", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "objecter_timeout": "10.000000", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "objectstore_blackhole": "false", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "openssl_conf": "", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_agent_delay_time": "5.000000", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_agent_hist_halflife": "1000", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_agent_max_low_ops": "2", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_agent_max_ops": "4", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_agent_min_evict_effort": "0.100000", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_agent_quantize_effort": "0.100000", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_agent_slop": "0.020000", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_aggregated_slow_ops_logging": "true", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_allow_recovery_below_min_size": "true", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_asio_thread_count": "2", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_async_recovery_min_cost": "100", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_auto_mark_unfound_lost": "false", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_backfill_retry_interval": "30.000000", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_backfill_scan_max": "512", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_backfill_scan_min": "64", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_backoff_on_degraded": "false", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_backoff_on_peering": "false", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_backoff_on_unfound": "true", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_beacon_report_interval": "300", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_bench_duration": "30", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_bench_large_size_max_throughput": "104857600", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_bench_max_block_size": "67108864", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_bench_small_size_max_iops": "100", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_blkin_trace_all": "false", 2026-02-25T08:41:23.418 INFO:teuthology.orchestra.run.trial039.stdout: "osd_blocked_scrub_grace_period": "120", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_check_for_log_corruption": "false", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_check_max_object_name_len_on_startup": "true", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_class_update_on_start": "true", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_client_message_cap": "256", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_client_message_size_cap": "524288000", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_client_op_priority": "63", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_client_watch_timeout": "30", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_command_max_records": "256", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_compact_on_start": "false", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_copyfrom_max_chunk": "8388608", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_crush_chooseleaf_type": "0", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_crush_initial_weight": "-1.000000", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_crush_update_on_start": "true", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_crush_update_weight_set": "true", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_drop_ping_duration": "0", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_drop_ping_probability": "0.000000", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_feed_pullee": "-1", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_inject_copyfrom_error": "false", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_misdirected_ops": "false", 2026-02-25T08:41:23.419 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_no_acting_change": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_no_purge_strays": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_op_order": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_pg_log_writeout": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_pretend_recovery_active": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_random_push_read_error": "0.000000", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_shutdown": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_trim_objects": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_verify_cached_snaps": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_verify_missing_on_start": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_verify_snaps": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_debug_verify_stray_on_activate": "false", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_deep_scrub_interval": "604800.000000", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_deep_scrub_interval_cv": "0.200000", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_deep_scrub_keys": "1024", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_deep_scrub_stride": "4194304", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_default_data_pool_replay_window": "5", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_default_notify_timeout": "30", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_delete_sleep": "0.000000", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_delete_sleep_hdd": "5.000000", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_delete_sleep_hybrid": "1.000000", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_delete_sleep_ssd": "1.000000", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_discard_disconnected_ops": "true", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_ec_partial_reads": "true", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_enable_op_tracker": "true", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2026-02-25T08:41:23.420 INFO:teuthology.orchestra.run.trial039.stdout: "osd_extblkdev_plugins": "vdo", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_failsafe_full_ratio": "0.970000", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_fast_fail_on_connection_refused": "true", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_fast_info": "true", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_fast_shutdown": "true", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_fast_shutdown_notify_mon": "true", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_fast_shutdown_timeout": "15", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_find_best_info_ignore_history_les": "false", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_force_auth_primary_missing_objects": "100", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_function_tracing": "false", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_heartbeat_grace": "20", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_heartbeat_interval": "6", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_heartbeat_min_peers": "10", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_heartbeat_min_size": "2000", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_heartbeat_stale": "600", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_hit_set_max_size": "100000", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_hit_set_min_size": "1000", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_hit_set_namespace": ".ceph-internal", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_ignore_stale_divergent_priors": "false", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_inject_failure_on_pg_removal": "false", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_journal_flush_on_shutdown": "true", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_journal_size": "5120", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_kill_backfill_at": "0", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_loop_before_reset_tphandle": "64", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_map_cache_size": "50", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_map_dedup": "true", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_map_message_max": "40", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_map_message_max_bytes": "10485760", 2026-02-25T08:41:23.421 INFO:teuthology.orchestra.run.trial039.stdout: "osd_map_share_max_epochs": "40", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_attr_name_len": "100", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_attr_size": "0", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_backfills": "1", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_markdown_count": "5", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_markdown_period": "600", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_object_name_len": "2048", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_object_namespace_len": "256", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_object_size": "134217728", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_omap_entries_per_request": "1024", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_pg_blocked_by": "16", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_pg_log_entries": "2", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_pgls": "1024", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_push_cost": "8388608", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_push_objects": "10", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_scrubs": "3", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_trimming_pgs": "2", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_write_op_reply_len": "64", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_max_write_size": "90", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_iops_capacity_low_threshold_hdd": "50.000000", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_iops_capacity_low_threshold_ssd": "1000.000000", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_override_recovery_settings": "false", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_profile": "balanced", 2026-02-25T08:41:23.422 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_scheduler_client_wgt": "1", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mclock_skip_benchmark": "false", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_memory_base": "805306368", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_memory_cache_min": "134217728", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_memory_cache_resize_interval": "1.000000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_memory_expected_fragmentation": "0.150000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_memory_target": "4294967296", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_memory_target_autotune": "false", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_min_pg_log_entries": "1", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_min_recovery_priority": "0", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mon_heartbeat_interval": "30", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mon_report_interval": "5", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mon_report_max_in_flight": "2", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_mon_shutdown_timeout": "5.000000", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_num_cache_shards": "32", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_num_op_tracker_shard": "32", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_numa_auto_affinity": "true", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_numa_node": "-1", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_numa_prefer_iface": "true", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_object_clean_region_max_num_intervals": "10", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_objecter_finishers": "1", 2026-02-25T08:41:23.423 INFO:teuthology.orchestra.run.trial039.stdout: "osd_objectstore": "bluestore", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_objectstore_fuse": "false", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_objectstore_ideal_list_max": "64", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_objectstore_tracing": "false", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_complaint_time": "30.000000", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_history_duration": "600", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_history_size": "20", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_history_slow_op_size": "20", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_log_threshold": "5", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_num_shards": "0", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_num_shards_hdd": "1", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_num_shards_ssd": "8", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_num_threads_per_shard": "0", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_num_threads_per_shard_hdd": "5", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_pq_min_cost": "65536", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_queue": "mclock_scheduler", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_queue_cut_off": "high", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_thread_suicide_timeout": "150", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_op_thread_timeout": "15", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_open_classes_on_start": "true", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_os_flags": "0", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_peering_op_priority": "255", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_delete_cost": "1048576", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_delete_priority": "5", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_log_dups_tracked": "3000", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_log_trim_max": "10000", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_log_trim_min": "0", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_object_context_cache_count": "64", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2026-02-25T08:41:23.424 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_cache_min_evict_age": "0", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_cache_min_flush_age": "0", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_crimson": "true", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_crush_rule": "-1", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_ec_fast_read": "false", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_erasure_code_profile": "plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_flag_bulk": "false", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_flag_ec_optimizations": "false", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_flag_hashpspool": "true", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_flag_nodelete": "false", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_flag_nopgchange": "false", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_flag_nosizechange": "false", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_flags": "0", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_min_size": "2", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_pg_num": "32", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_pgp_num": "0", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_read_ratio": "70", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_size": "3", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_default_type": "replicated", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_erasure_code_stripe_unit": "0", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_pool_use_gmt_hitset": "true", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_push_per_object_cost": "1000", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_read_ec_check_for_errors": "false", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recover_clone_overlap": "true", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recover_clone_overlap_limit": "10", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_cost": "20971520", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_delay_start": "0.000000", 2026-02-25T08:41:23.425 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_max_active": "0", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_max_active_hdd": "3", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_max_active_ssd": "10", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_max_chunk": "8388608", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_max_single_start": "1", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_op_priority": "3", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_op_warn_multiple": "16", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_priority": "5", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_retry_interval": "30.000000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_sleep": "0.000000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_sleep_degraded": "0.000000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_sleep_degraded_hdd": "0.100000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_sleep_degraded_hybrid": "0.025000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_sleep_degraded_ssd": "0.000000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_sleep_hdd": "0.100000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_recovery_sleep_ssd": "0.000000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_requested_scrub_priority": "5", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_rollback_to_cluster_snap": "", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_auto_repair": "false", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_auto_repair_num_errors": "5", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_backoff_ratio": "0.660000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_begin_hour": "0", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_begin_week_day": "0", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_chunk_max": "15", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_chunk_min": "5", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_cost": "52428800", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_disable_reservation_queuing": "false", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_during_recovery": "false", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_end_hour": "0", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_end_week_day": "0", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_event_cost": "4096", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_extended_sleep": "0.000000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_invalid_stats": "true", 2026-02-25T08:41:23.426 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_load_threshold": "10.000000", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_max_interval": "604800.000000", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_max_preemptions": "5", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_min_interval": "86400.000000", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_priority": "5", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_retry_after_noscrub": "60", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_retry_delay": "30", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_retry_new_interval": "10", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_retry_pg_state": "60", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_retry_trimming": "10", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_scrub_sleep": "0.000000", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_shallow_scrub_chunk_max": "100", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_shallow_scrub_chunk_min": "50", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_shutdown_pgref_assert": "false", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_skip_check_past_interval_bounds": "false", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_skip_data_digest": "false", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_smart_report_timeout": "5", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_snap_trim_cost": "1048576", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_snap_trim_priority": "5", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_snap_trim_sleep": "0.000000", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_stats_update_period_not_scrubbing": "120", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_stats_update_period_scrubbing": "15", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_target_transaction_size": "30", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tier_default_cache_hit_set_count": "4", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tier_default_cache_mode": "writeback", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tier_promote_max_objects_sec": "25", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_tracing": "false", 2026-02-25T08:41:23.427 INFO:teuthology.orchestra.run.trial039.stdout: "osd_use_stale_snap": "false", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "osdc_blkin_trace_all": "false", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_kill_at": "0", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_max_join_drift": "10", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_min": "500", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_min_wait": "0.050000", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_propose_interval": "1.000000", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_service_trim_max": "500", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_service_trim_max_multiplier": "20", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_service_trim_min": "10", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_stash_full_interval": "25", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_trim_max": "500", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "paxos_trim_min": "250", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "perf": "true", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "plugin_crypto_accelerator": "crypto_isal", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "plugin_dir": "/usr/lib64/ceph", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "pool_availability_update_interval": "1.000000", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "public_addr": "-", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "public_addrv": "", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "public_bind_addr": "-", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "public_network": "", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "public_network_interface": "", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "qat_compressor_busy_polling": "false", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "qat_compressor_enabled": "false", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "qat_compressor_session_max_number": "256", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rados_mon_op_timeout": "0", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rados_osd_op_timeout": "0", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rados_replica_read_policy": "default", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rados_replica_read_policy_on_objclass": "false", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rados_tracing": "false", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_atime_update_interval": "60", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_balance_parent_reads": "false", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_balance_snap_reads": "false", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_blkin_trace_all": "false", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_blocklist_expire_seconds": "0", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_blocklist_on_break_lock": "true", 2026-02-25T08:41:23.428 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_cache": "true", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_cache_block_writes_upfront": "false", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_cache_max_dirty": "25165824", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_cache_max_dirty_age": "1.000000", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_cache_max_dirty_object": "0", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_cache_policy": "writearound", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_cache_size": "33554432", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_cache_target_dirty": "16777216", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_cache_writethrough_until_flush": "true", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_clone_copy_on_read": "false", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_compression_hint": "none", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_concurrent_management_ops": "10", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_config_pool_override_update_timestamp": "0", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_default_clone_format": "auto", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_default_data_pool": "", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_default_features": "61", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_default_format": "2", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_default_map_options": "", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_default_order": "22", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_default_pool": "rbd", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_default_stripe_count": "0", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_default_stripe_unit": "0", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_disable_zero_copy_writes": "true", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_discard_granularity_bytes": "65536", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_discard_on_zeroed_write_same": "true", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_enable_alloc_hint": "true", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_io_scheduler": "simple", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_commit_age": "5.000000", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_max_payload_bytes": "16384", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_object_flush_age": "0.000000", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_object_flush_bytes": "1048576", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_object_flush_interval": "0", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_order": "24", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_pool": "", 2026-02-25T08:41:23.429 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_journal_splay_width": "4", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_localize_parent_reads": "false", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_localize_snap_reads": "false", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_image_policy_type": "simple", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_image_state_check_interval": "30", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_memory_autotune": "true", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_memory_base": "805306368", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_memory_cache_min": "134217728", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_memory_target": "4294967296", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_perf_stats_prio": "5", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirroring_delete_delay": "0", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirroring_replay_delay": "0", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_move_to_trash_on_remove": "false", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_mtime_update_interval": "60", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_non_blocking_aio": "true", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_op_thread_timeout": "60", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_op_threads": "1", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_parent_cache_enabled": "false", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_persistent_cache_mode": "disabled", 2026-02-25T08:41:23.430 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_persistent_cache_path": "/tmp", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_persistent_cache_size": "1073741824", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_plugins": "", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_bps_burst": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_bps_burst_seconds": "1", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_bps_limit": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_exclude_ops": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_iops_burst": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_iops_burst_seconds": "1", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_iops_limit": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_read_bps_burst": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_read_bps_limit": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_read_iops_burst": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_read_iops_limit": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_schedule_tick_min": "50", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_write_bps_burst": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_write_bps_limit": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_write_iops_burst": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_qos_write_iops_limit": "0", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_quiesce_notification_attempts": "10", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_read_from_replica_policy": "default", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_readahead_max_bytes": "524288", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_readahead_trigger_requests": "10", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_request_timed_out_seconds": "30", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_skip_partial_discard": "true", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_tracing": "false", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_validate_names": "true", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rbd_validate_pool": "true", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "restapi_base_url": "", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "restapi_log_level": "", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_account_default_quota_max_objects": "-1", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_account_default_quota_max_size": "-1", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_acl_grants_max_num": "100", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_admin_entry": "admin", 2026-02-25T08:41:23.431 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_asio_assert_yielding": "false", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_backend_store": "rados", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_barbican_url": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_beast_enable_async": "true", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_counters_cache": "false", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_counters_cache_size": "10000", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_default_quota_max_size": "-1", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_eexist_override": "false", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_index_max_aio": "128", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_logging_obj_roll_time": "300", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_persistent_notif_num_shards": "11", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_quota_cache_size": "10000", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_quota_ttl": "600", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_bucket_sync_spawn_window": "20", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_cache_enabled": "true", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_cache_expiry_interval": "900", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_cache_lru_size": "25000", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_config_store": "rados", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_content_length_compat": "false", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_copy_obj_progress": "true", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_cors_rules_max_num": "100", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_cross_domain_policy": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_default_encryption_key": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_kmip_addr": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_kmip_ca_path": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_kmip_client_cert": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_kmip_client_key": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_kmip_kms_key_template": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_kmip_password": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_kmip_username": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_require_ssl": "true", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_backend": "vault", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2026-02-25T08:41:23.432 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_suppress_logs": "true", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_vault_addr": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_vault_auth": "token", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_vault_namespace": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_vault_prefix": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_vault_secret_engine": "transit", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_vault_ssl_cacert": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_vault_token_file": "", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_crypt_vault_verify_ssl": "true", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_curl_buffersize": "524288", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_curl_low_speed_limit": "1024", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_curl_low_speed_time": "300", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_curl_tcp_keepalive": "0", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_curl_wait_timeout_ms": "1000", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d3n_l1_fadvise": "4", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d3n_libaio_aio_num": "64", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d3n_libaio_aio_threads": "20", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_backend_address": "127.0.0.1:6379", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_cache_cleaning_interval": "1000", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_l1_datacache_disk_reserve": "1073741824", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2026-02-25T08:41:23.433 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_l1_fadvise": "4", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_l1_write_open_flags": "4096", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_libaio_aio_num": "64", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_libaio_aio_threads": "20", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_d4n_local_rgw_address": "127.0.0.1:8000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_data_log_changes_size": "1000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_data_log_num_shards": "128", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_data_log_window": "30", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_data_notify_interval_msec": "0", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_data_sync_poll_interval": "20", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_data_sync_spawn_window": "20", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_debug_inject_set_olh_err": "0", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_default_data_log_backing": "fifo", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_default_realm_info_oid": "default.realm", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_default_region_info_oid": "default.region", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_default_zone_info_oid": "default.zone", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_defer_to_bucket_acls": "", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_delete_multi_obj_max_num": "1000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_disable_s3select": "false", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_admin_lim": "0.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_admin_res": "100.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_auth_lim": "0.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_auth_res": "200.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_data_lim": "0.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_data_res": "500.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_data_wgt": "500.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_metadata_res": "500.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dns_name": "", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dns_s3website_name": "", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dynamic_resharding": "true", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enable_gc_threads": "true", 2026-02-25T08:41:23.434 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enable_jwks_url_verification": "false", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enable_lc_threads": "true", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enable_mdsearch": "true", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enable_ops_log": "false", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enable_quota_threads": "true", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enable_restore_threads": "true", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enable_static_website": "false", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enable_usage_log": "false", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_enforce_swift_acls": "true", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_exit_timeout_secs": "120", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_expose_bucket": "false", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_extended_http_attrs": "", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_filter": "none", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_frontends": "beast port=7480", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_gc_max_concurrent_io": "10", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_gc_max_deferred": "50", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_gc_max_objs": "32", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_gc_max_queue_size": "134213632", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_gc_max_trim_chunk": "16", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_gc_obj_min_wait": "7200", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_gc_processor_max_time": "3600", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_gc_processor_period": "3600", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_get_obj_max_req_size": "4194304", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_get_obj_window_size": "16777216", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_graceful_stop": "false", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_healthcheck_disabling_path": "", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_http_notif_connection_timeout": "5", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_http_notif_max_inflight": "8192", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_http_notif_message_timeout": "10", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ignore_get_invalid_range": "false", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_init_timeout": "300", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_kafka_connection_idle": "300", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_kafka_message_timeout": "5000", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_kafka_sleep_timeout": "10", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_accepted_admin_roles": "", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_accepted_reader_roles": "", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_admin_domain": "", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_admin_password": "", 2026-02-25T08:41:23.435 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_admin_password_path": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_admin_project": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_admin_tenant": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_admin_user": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_barbican_domain": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_barbican_password": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_barbican_project": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_barbican_tenant": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_barbican_user": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_implicit_tenants": "false", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_service_token_enabled": "false", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_token_cache_size": "10000", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_token_cache_ttl": "300", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_url": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_keystone_verify_ssl": "true", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lc_debug_interval": "-1", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lc_list_cnt": "1000", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lc_lock_max_time": "90", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lc_max_objs": "32", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lc_max_rules": "1000", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lc_max_worker": "3", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lc_max_wp_worker": "128", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lc_ordered_list_threshold": "500", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lc_thread_delay": "0", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ldap_dnattr": "uid", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ldap_searchfilter": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ldap_uri": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lfuda_sync_frequency": "60", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_list_bucket_min_readahead": "1000", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_list_buckets_max_chunk": "1000", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_log_http_headers": "", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_log_nonexistent_bucket": "false", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_log_object_name_utc": "false", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lua_enable": "true", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lua_max_memory_per_state": "128000", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_lua_max_runtime_per_state": "1000", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2026-02-25T08:41:23.436 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_attr_name_len": "0", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_attr_size": "0", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_attrs_num_in_req": "0", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_chunk_size": "4194304", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_concurrent_requests": "1024", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_control_aio": "8", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_dynamic_shards": "1999", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_listing_results": "5000", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_notify_retries": "10", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_objs_per_shard": "100000", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_put_param_size": "1048576", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_put_size": "5368709120", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_max_slo_entries": "1000", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_md_log_max_shards": "64", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_md_notify_interval_msec": "200", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_meta_sync_poll_interval": "20", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_meta_sync_spawn_window": "20", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_mime_types_file": "/etc/mime.types", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_mp_lock_max_time": "600", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_multi_obj_del_max_aio": "16", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_multipart_min_part_size": "5242880", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_multipart_part_upload_limit": "10000", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_fhcache_partitions": "3", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_fhcache_size": "2017", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_frontends": "rgw-nfs", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_lru_lanes": "5", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_max_gc": "300", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_namespace_expire_secs": "300", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_run_gc_threads": "false", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_run_lc_threads": "false", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_run_quota_threads": "false", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_run_restore_threads": "false", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_run_sync_thread": "false", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_s3_fast_attrs": "false", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_nfs_write_completion_interval_s": "10", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_num_async_rados_threads": "32", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_num_control_oids": "8", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_numa_node": "-1", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_obj_stripe_size": "4194304", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_obj_tombstone_cache_size": "1000", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_objexp_chunk_size": "100", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_objexp_gc_interval": "600", 2026-02-25T08:41:23.437 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_objexp_hints_num_shards": "127", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_olh_pending_timeout_sec": "3600", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_op_thread_suicide_timeout": "0", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_op_thread_timeout": "600", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_op_tracing": "false", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_opa_token": "", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_opa_url": "", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_opa_verify_ssl": "true", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ops_log_data_backlog": "5242880", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ops_log_rados": "false", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ops_log_socket_path": "", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_override_bucket_index_max_shards": "0", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_parquet_buffer_size": "16777216", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_period_push_interval": "2.000000", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_period_push_interval_max": "30.000000", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_period_root_pool": ".rgw.root", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_policy_reject_invalid_principals": "true", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_posix_cache_lanes": "3", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_posix_cache_lmdb_count": "3", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_posix_cache_max_buckets": "100", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_posix_cache_partitions": "3", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_posix_userdb_dir": "/var/lib/ceph/radosgw", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_print_continue": "true", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_print_prohibited_content_length": "false", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_put_obj_max_window_size": "67108864", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_put_obj_min_window_size": "16777216", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_rados_pool_recovery_priority": "5", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_rados_tracing": "false", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_ratelimit_interval": "60", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_realm": "", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_realm_id": "", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_realm_root_pool": ".rgw.root", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_redis_connection_pool_size": "512", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_region": "", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_region_root_pool": ".rgw.root", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_relaxed_region_enforcement": "false", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_relaxed_s3_bucket_names": "false", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2026-02-25T08:41:23.438 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_request_uri": "", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_reshard_batch_size": "64", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_reshard_bucket_lock_duration": "360", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_reshard_debug_interval": "-1", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_reshard_max_aio": "128", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_reshard_num_logs": "16", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_reshard_progress_judge_interval": "120", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_reshard_thread_interval": "600", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_reshardlog_threshold": "30000", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_resolve_cname": "false", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_rest_getusage_op_compat": "false", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_restore_debug_interval": "-1", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_restore_lock_max_time": "90", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_restore_max_objs": "32", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_restore_processor_period": "900", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_run_sync_thread": "true", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_s3_auth_disable_signature_url": "false", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_s3_auth_order": "sts, external, local", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_s3_auth_use_keystone": "false", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_s3_auth_use_ldap": "false", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_s3_auth_use_rados": "true", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_s3_auth_use_sts": "false", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_s3_client_max_sig_ver": "-1", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_s3_success_create_obj_status": "0", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_safe_max_objects_per_shard": "102400", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_scheduler_type": "throttler", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_script_uri": "", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_service_provider_name": "", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_shard_warning_threshold": "90.000000", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sts_client_id": "", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sts_client_secret": "", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sts_entry": "sts", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sts_key": "", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sts_max_session_duration": "43200", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sts_min_session_duration": "900", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sts_token_introspection_url": "", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_account_in_url": "false", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_auth_entry": "auth", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_auth_url": "", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_custom_header": "", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_enforce_content_length": "false", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_need_stats": "true", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_tenant_name": "", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_token_expiration": "86400", 2026-02-25T08:41:23.439 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_url": "", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_url_prefix": "swift", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_swift_versioning_enabled": "false", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_lease_period": "120", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_log_trim_interval": "1200", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_log_trim_max_buckets": "16", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_obj_etag_verify": "false", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_trace_history_size": "4096", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_trace_per_node_log_size": "32", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_thread_pool_size": "512", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_topic_persistency_max_retries": "0", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_topic_persistency_sleep_duration": "0", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_topic_persistency_time_to_live": "0", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_topic_require_publish_policy": "false", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_torrent_comment": "", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_torrent_createby": "", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_torrent_encoding": "", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_torrent_flag": "false", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_torrent_max_size": "5368709120", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_torrent_origin": "", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_torrent_sha_unit": "524288", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_torrent_tracker": "", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_trust_forwarded_https": "false", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_usage_log_flush_threshold": "1024", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_usage_log_key_transition": "true", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_usage_log_tick_interval": "30", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_usage_max_shards": "32", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_usage_max_user_shards": "1", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_use_opa_authz": "false", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_counters_cache": "false", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_counters_cache_size": "10000", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_default_quota_max_objects": "-1", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_default_quota_max_size": "-1", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_max_buckets": "1000", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_policies_max_num": "100", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_quota_sync_idle_users": "false", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_quota_sync_interval": "86400", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_quota_sync_wait_time": "86400", 2026-02-25T08:41:23.440 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_user_unique_email": "true", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_verify_ssl": "true", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_website_routing_rules_max_num": "50", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_zone": "", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_zone_id": "", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_zone_root_pool": ".rgw.root", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_zonegroup": "", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_zonegroup_id": "", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rgwlc_auto_session_clear": "true", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rgwlc_skip_bucket_step": "false", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_block_size": "4096", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_bloom_bits_per_key": "20", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_cache_row_ratio": "0.000000", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_cache_shard_bits": "4", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_cache_size": "536870912", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_cache_type": "binned_lru", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_cf_compact_on_deletion": "true", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_collect_compaction_stats": "false", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_collect_extended_stats": "false", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_collect_memory_stats": "false", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_delete_range_threshold": "1048576", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_index_type": "binary_search", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_log_to_ceph_log": "true", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_metadata_block_size": "4096", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_partition_filters": "false", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_perf": "false", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rotating_keys_bootstrap_timeout": "30", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "rotating_keys_renewal_timeout": "10", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "run_dir": "/var/run/ceph", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_block_create": "true", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_cachepin_2q_in_ratio": "0.500000", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_cachepin_2q_out_ratio": "0.500000", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_cachepin_size_pershard": "2147483648", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_cachepin_type": "LRU", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_cbjournal_size": "5368709120", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_cold_tier_generations": "3", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_data_delta_based_overwrite": "0", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_default_max_object_size": "16777216", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_default_object_metadata_reservation": "16777216", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_device_size": "53687091200", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_disable_end_to_end_data_protection": "true", 2026-02-25T08:41:23.441 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_full_integrity_check": "false", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_hot_tier_generations": "5", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_journal_batch_capacity": "16", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_journal_batch_flush_size": "16777216", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_journal_iodepth_limit": "5", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_main_device_type": "SSD", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_max_concurrent_transactions": "128", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_max_data_allocation_size": "0", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_segment_cleaner_gc_formula": "cost_benefit", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "seastore_segment_size": "67108864", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "service_unique_id": "", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "set_keepcaps": "false", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "setgroup": "", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "setuser": "", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "setuser_match_path": "", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "subv_metrics_window_interval": "30", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "target_max_misplaced_ratio": "0.050000", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "thp": "false", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "threadpool_default_timeout": "60", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "threadpool_empty_queue_max_wait": "2", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "throttler_perf_counter": "true", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "tmp_dir": "/tmp", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "uadk_compressor_enabled": "false", 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout: "uadk_wd_sync_ctx_num": "2" 2026-02-25T08:41:23.442 INFO:teuthology.orchestra.run.trial039.stdout:} 2026-02-25T08:41:23.443 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2026-02-25T08:41:23.517 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:23.514+0000 7fc3bfe3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.520 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:23.518+0000 7fc3bfe3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.529 INFO:teuthology.orchestra.run.trial039.stdout:{} 2026-02-25T08:41:23.529 INFO:teuthology.orchestra.run.trial039.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2026-02-25T08:41:23.535 DEBUG:teuthology.orchestra.run.trial091:> type ceph-objectstore-tool 2026-02-25T08:41:23.554 INFO:teuthology.orchestra.run.trial091.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-25T08:41:23.554 DEBUG:teuthology.orchestra.run.trial039:> type ceph-objectstore-tool 2026-02-25T08:41:23.571 INFO:teuthology.orchestra.run.trial039.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-25T08:41:23.571 INFO:teuthology.run_tasks:Running task rados... 2026-02-25T08:41:23.594 INFO:tasks.rados:Beginning rados... 2026-02-25T08:41:23.594 INFO:tasks.rados:config is {'balance_reads': True, 'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2026-02-25T08:41:23.594 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}} 2026-02-25T08:41:23.594 INFO:tasks.rados:config is {'balance_reads': True, 'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2026-02-25T08:41:23.595 DEBUG:teuthology.run_tasks:Unwinding manager rados 2026-02-25T08:41:23.610 INFO:tasks.rados:joining rados 2026-02-25T08:41:23.610 INFO:tasks.thrashosds.thrasher:starting do_thrash 2026-02-25T08:41:23.610 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-25T08:41:23.610 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T08:41:23.611 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3] 2026-02-25T08:41:23.611 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-25T08:41:23.611 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:41:23.611 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2026-02-25T08:41:23.612 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:23.612 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2026-02-25T08:41:23.612 INFO:tasks.rados:starting run 0 out of 1 2026-02-25T08:41:23.612 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2026-02-25T08:41:23.613 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2026-02-25T08:41:23.670 INFO:tasks.ceph.osd.2:Stopped 2026-02-25T08:41:23.689 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:23.686+0000 7f3eef01a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.692 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:23.690+0000 7f3eef01a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.693 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:23.687+0000 7f805b0b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.696 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:23.694+0000 7f805b0b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.107 INFO:teuthology.orchestra.run.trial039.stderr:pool 'unique_pool_0' created 2026-02-25T08:41:24.121 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-25T08:41:24.240 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:24.233+0000 7ffa0e8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:24.242 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:24.240+0000 7ffa0e8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:25.111 INFO:teuthology.orchestra.run.trial039.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-25T08:41:26.127 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pct_update_delay 5 2026-02-25T08:41:26.206 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:26.200+0000 7f2bfd10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.209 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:26.207+0000 7f2bfd10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.118 INFO:teuthology.orchestra.run.trial039.stderr:set pool 3 pct_update_delay to 5 2026-02-25T08:41:27.134 DEBUG:teuthology.orchestra.run.trial039:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2026-02-25T08:41:27.182 INFO:tasks.rados.rados.0.trial039.stdout:adding op weight read -> 100 2026-02-25T08:41:27.182 INFO:tasks.rados.rados.0.trial039.stdout:adding op weight write -> 50 2026-02-25T08:41:27.182 INFO:tasks.rados.rados.0.trial039.stdout:adding op weight delete -> 50 2026-02-25T08:41:27.182 INFO:tasks.rados.rados.0.trial039.stdout:adding op weight snap_create -> 50 2026-02-25T08:41:27.182 INFO:tasks.rados.rados.0.trial039.stdout:adding op weight snap_remove -> 50 2026-02-25T08:41:27.182 INFO:tasks.rados.rados.0.trial039.stdout:adding op weight write_excl -> 50 2026-02-25T08:41:27.182 INFO:tasks.rados.rados.0.trial039.stdout:ceph version 20.3.0-5338-g6aeb6975 (6aeb6975bc9a4790a399c9cc70dba9635d2a7819) tentacle (dev - Debug) 2026-02-25T08:41:27.183 INFO:tasks.rados.rados.0.trial039.stdout:Configuration: 2026-02-25T08:41:27.183 INFO:tasks.rados.rados.0.trial039.stdout: Number of operations: 4000 2026-02-25T08:41:27.183 INFO:tasks.rados.rados.0.trial039.stdout: Number of objects: 50 2026-02-25T08:41:27.183 INFO:tasks.rados.rados.0.trial039.stdout: Max in flight operations: 16 2026-02-25T08:41:27.183 INFO:tasks.rados.rados.0.trial039.stdout: Object size (in bytes): 4000000 2026-02-25T08:41:27.183 INFO:tasks.rados.rados.0.trial039.stdout: Write stride min: 400000 2026-02-25T08:41:27.183 INFO:tasks.rados.rados.0.trial039.stdout: Write stride max: 800000 2026-02-25T08:41:28.123 INFO:tasks.rados.rados.0.trial039.stdout:1: write initial oid 1 2026-02-25T08:41:28.123 INFO:tasks.rados.rados.0.trial039.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2026-02-25T08:41:28.126 INFO:tasks.rados.rados.0.trial039.stdout:1: writing trial03942481-1 from 416807 to 892056 tid 1 2026-02-25T08:41:28.131 INFO:tasks.rados.rados.0.trial039.stdout:1: writing trial03942481-1 from 1542129 to 2016807 tid 2 2026-02-25T08:41:28.131 INFO:tasks.rados.rados.0.trial039.stdout:2: write initial oid 2 2026-02-25T08:41:28.131 INFO:tasks.rados.rados.0.trial039.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2026-02-25T08:41:28.135 INFO:tasks.rados.rados.0.trial039.stdout:2: writing trial03942481-2 from 416807 to 892056 tid 1 2026-02-25T08:41:28.139 INFO:tasks.rados.rados.0.trial039.stdout:2: writing trial03942481-2 from 1542129 to 2016807 tid 2 2026-02-25T08:41:28.140 INFO:tasks.rados.rados.0.trial039.stdout:3: write initial oid 3 2026-02-25T08:41:28.140 INFO:tasks.rados.rados.0.trial039.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2026-02-25T08:41:28.143 INFO:tasks.rados.rados.0.trial039.stdout:3: writing trial03942481-3 from 433614 to 984112 tid 1 2026-02-25T08:41:28.147 INFO:tasks.rados.rados.0.trial039.stdout:3: writing trial03942481-3 from 1600611 to 2033614 tid 2 2026-02-25T08:41:28.147 INFO:tasks.rados.rados.0.trial039.stdout:4: write initial oid 4 2026-02-25T08:41:28.147 INFO:tasks.rados.rados.0.trial039.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2026-02-25T08:41:28.151 INFO:tasks.rados.rados.0.trial039.stdout:4: writing trial03942481-4 from 450421 to 1076168 tid 1 2026-02-25T08:41:28.155 INFO:tasks.rados.rados.0.trial039.stdout:4: writing trial03942481-4 from 1659093 to 2050421 tid 2 2026-02-25T08:41:28.155 INFO:tasks.rados.rados.0.trial039.stdout:5: write initial oid 5 2026-02-25T08:41:28.155 INFO:tasks.rados.rados.0.trial039.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2026-02-25T08:41:28.160 INFO:tasks.rados.rados.0.trial039.stdout:5: writing trial03942481-5 from 467228 to 1168224 tid 1 2026-02-25T08:41:28.164 INFO:tasks.rados.rados.0.trial039.stdout:5: writing trial03942481-5 from 1717575 to 2067228 tid 2 2026-02-25T08:41:28.164 INFO:tasks.rados.rados.0.trial039.stdout:6: write initial oid 6 2026-02-25T08:41:28.164 INFO:tasks.rados.rados.0.trial039.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2026-02-25T08:41:28.168 INFO:tasks.rados.rados.0.trial039.stdout:6: writing trial03942481-6 from 484035 to 1260280 tid 1 2026-02-25T08:41:28.172 INFO:tasks.rados.rados.0.trial039.stdout:6: writing trial03942481-6 from 2059704 to 2084035 tid 2 2026-02-25T08:41:28.172 INFO:tasks.rados.rados.0.trial039.stdout:7: write initial oid 7 2026-02-25T08:41:28.172 INFO:tasks.rados.rados.0.trial039.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2026-02-25T08:41:28.175 INFO:tasks.rados.rados.0.trial039.stdout:7: writing trial03942481-7 from 500842 to 952336 tid 1 2026-02-25T08:41:28.180 INFO:tasks.rados.rados.0.trial039.stdout:7: writing trial03942481-7 from 1718186 to 2100842 tid 2 2026-02-25T08:41:28.180 INFO:tasks.rados.rados.0.trial039.stdout:8: write initial oid 8 2026-02-25T08:41:28.180 INFO:tasks.rados.rados.0.trial039.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2026-02-25T08:41:28.184 INFO:tasks.rados.rados.0.trial039.stdout:8: writing trial03942481-8 from 517649 to 1044392 tid 1 2026-02-25T08:41:28.188 INFO:tasks.rados.rados.0.trial039.stdout:8: writing trial03942481-8 from 1776668 to 2117649 tid 2 2026-02-25T08:41:28.188 INFO:tasks.rados.rados.0.trial039.stdout:9: write initial oid 9 2026-02-25T08:41:28.188 INFO:tasks.rados.rados.0.trial039.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2026-02-25T08:41:28.193 INFO:tasks.rados.rados.0.trial039.stdout:9: writing trial03942481-9 from 534456 to 1252801 tid 1 2026-02-25T08:41:28.197 INFO:tasks.rados.rados.0.trial039.stdout:9: writing trial03942481-9 from 1951503 to 2134456 tid 2 2026-02-25T08:41:28.197 INFO:tasks.rados.rados.0.trial039.stdout:10: write initial oid 10 2026-02-25T08:41:28.197 INFO:tasks.rados.rados.0.trial039.stdout:3: finishing write tid 1 to trial03942481-3 2026-02-25T08:41:28.197 INFO:tasks.rados.rados.0.trial039.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2026-02-25T08:41:28.202 INFO:tasks.rados.rados.0.trial039.stdout:10: writing trial03942481-10 from 551263 to 1344857 tid 1 2026-02-25T08:41:28.205 INFO:tasks.rados.rados.0.trial039.stdout:10: writing trial03942481-10 from 1893632 to 2151263 tid 2 2026-02-25T08:41:28.205 INFO:tasks.rados.rados.0.trial039.stdout:11: write initial oid 11 2026-02-25T08:41:28.206 INFO:tasks.rados.rados.0.trial039.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2026-02-25T08:41:28.209 INFO:tasks.rados.rados.0.trial039.stdout:11: writing trial03942481-11 from 568070 to 1036913 tid 1 2026-02-25T08:41:28.214 INFO:tasks.rados.rados.0.trial039.stdout:11: writing trial03942481-11 from 1552114 to 2168070 tid 2 2026-02-25T08:41:28.214 INFO:tasks.rados.rados.0.trial039.stdout:12: write initial oid 12 2026-02-25T08:41:28.214 INFO:tasks.rados.rados.0.trial039.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2026-02-25T08:41:28.218 INFO:tasks.rados.rados.0.trial039.stdout:12: writing trial03942481-12 from 584877 to 1128969 tid 1 2026-02-25T08:41:28.222 INFO:tasks.rados.rados.0.trial039.stdout:12: writing trial03942481-12 from 1610596 to 2172599 tid 2 2026-02-25T08:41:28.222 INFO:tasks.rados.rados.0.trial039.stdout:13: write initial oid 13 2026-02-25T08:41:28.223 INFO:tasks.rados.rados.0.trial039.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2026-02-25T08:41:28.227 INFO:tasks.rados.rados.0.trial039.stdout:13: writing trial03942481-13 from 601684 to 1221025 tid 1 2026-02-25T08:41:28.231 INFO:tasks.rados.rados.0.trial039.stdout:13: writing trial03942481-13 from 1669078 to 2201684 tid 2 2026-02-25T08:41:28.231 INFO:tasks.rados.rados.0.trial039.stdout:14: write initial oid 14 2026-02-25T08:41:28.231 INFO:tasks.rados.rados.0.trial039.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2026-02-25T08:41:28.236 INFO:tasks.rados.rados.0.trial039.stdout:14: writing trial03942481-14 from 618491 to 1313081 tid 1 2026-02-25T08:41:28.240 INFO:tasks.rados.rados.0.trial039.stdout:14: writing trial03942481-14 from 2011207 to 2218491 tid 2 2026-02-25T08:41:28.240 INFO:tasks.rados.rados.0.trial039.stdout:15: write initial oid 15 2026-02-25T08:41:28.240 INFO:tasks.rados.rados.0.trial039.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2026-02-25T08:41:28.245 INFO:tasks.rados.rados.0.trial039.stdout:15: writing trial03942481-15 from 635298 to 1405137 tid 1 2026-02-25T08:41:28.248 INFO:tasks.rados.rados.0.trial039.stdout:15: writing trial03942481-15 from 2069689 to 2235298 tid 2 2026-02-25T08:41:28.248 INFO:tasks.rados.rados.0.trial039.stdout:16: write initial oid 16 2026-02-25T08:41:28.249 INFO:tasks.rados.rados.0.trial039.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2026-02-25T08:41:28.253 INFO:tasks.rados.rados.0.trial039.stdout:16: writing trial03942481-16 from 652105 to 1097193 tid 1 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout:16: writing trial03942481-16 from 1728171 to 2181159 tid 2 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout:3: finishing write tid 2 to trial03942481-3 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout:5: finishing write tid 1 to trial03942481-5 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout:3: finishing write tid 3 to trial03942481-3 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout:3: finishing write tid 4 to trial03942481-3 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout:3: oid 3 updating version 0 to 1 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout:3: oid 3 updating version 1 to 3 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout:3: oid 3 version 3 is already newer than 3 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2026-02-25T08:41:28.258 INFO:tasks.rados.rados.0.trial039.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2026-02-25T08:41:28.259 INFO:tasks.rados.rados.0.trial039.stdout:6: finishing write tid 1 to trial03942481-6 2026-02-25T08:41:28.259 INFO:tasks.rados.rados.0.trial039.stdout:3: done (15 left) 2026-02-25T08:41:28.259 INFO:tasks.rados.rados.0.trial039.stdout:17: write initial oid 17 2026-02-25T08:41:28.259 INFO:tasks.rados.rados.0.trial039.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2026-02-25T08:41:28.262 INFO:tasks.rados.rados.0.trial039.stdout:17: writing trial03942481-17 from 668912 to 1305602 tid 1 2026-02-25T08:41:28.266 INFO:tasks.rados.rados.0.trial039.stdout:17: writing trial03942481-17 from 1903006 to 2268912 tid 2 2026-02-25T08:41:28.266 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:5: finishing write tid 2 to trial03942481-5 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:5: finishing write tid 3 to trial03942481-5 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:5: finishing write tid 4 to trial03942481-5 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:5: oid 5 updating version 0 to 3 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:5: oid 5 version 3 is already newer than 1 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:5: oid 5 version 3 is already newer than 3 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:6: finishing write tid 2 to trial03942481-6 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:5: done (15 left) 2026-02-25T08:41:28.267 INFO:tasks.rados.rados.0.trial039.stdout:18: write initial oid 18 2026-02-25T08:41:28.268 INFO:tasks.rados.rados.0.trial039.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2026-02-25T08:41:28.271 INFO:tasks.rados.rados.0.trial039.stdout:18: writing trial03942481-18 from 685719 to 1397658 tid 1 2026-02-25T08:41:28.275 INFO:tasks.rados.rados.0.trial039.stdout:18: writing trial03942481-18 from 1845135 to 2285719 tid 2 2026-02-25T08:41:28.275 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:28.275 INFO:tasks.rados.rados.0.trial039.stdout:6: finishing write tid 3 to trial03942481-6 2026-02-25T08:41:28.275 INFO:tasks.rados.rados.0.trial039.stdout:6: finishing write tid 4 to trial03942481-6 2026-02-25T08:41:28.276 INFO:tasks.rados.rados.0.trial039.stdout:6: oid 6 updating version 0 to 4 2026-02-25T08:41:28.276 INFO:tasks.rados.rados.0.trial039.stdout:6: oid 6 updating version 4 to 6 2026-02-25T08:41:28.276 INFO:tasks.rados.rados.0.trial039.stdout:6: oid 6 version 6 is already newer than 6 2026-02-25T08:41:28.276 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2026-02-25T08:41:28.276 INFO:tasks.rados.rados.0.trial039.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:41:28.276 INFO:tasks.rados.rados.0.trial039.stdout:6: done (15 left) 2026-02-25T08:41:28.276 INFO:tasks.rados.rados.0.trial039.stdout:19: write initial oid 19 2026-02-25T08:41:28.276 INFO:tasks.rados.rados.0.trial039.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2026-02-25T08:41:28.281 INFO:tasks.rados.rados.0.trial039.stdout:19: writing trial03942481-19 from 702526 to 1489714 tid 1 2026-02-25T08:41:28.284 INFO:tasks.rados.rados.0.trial039.stdout:19: writing trial03942481-19 from 1903617 to 2302526 tid 2 2026-02-25T08:41:28.284 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:28.298 INFO:tasks.rados.rados.0.trial039.stdout:19: finishing write tid 1 to trial03942481-19 2026-02-25T08:41:28.311 INFO:tasks.rados.rados.0.trial039.stdout:19: finishing write tid 2 to trial03942481-19 2026-02-25T08:41:28.315 INFO:tasks.rados.rados.0.trial039.stdout:19: finishing write tid 3 to trial03942481-19 2026-02-25T08:41:28.317 INFO:tasks.rados.rados.0.trial039.stdout:19: finishing write tid 4 to trial03942481-19 2026-02-25T08:41:28.317 INFO:tasks.rados.rados.0.trial039.stdout:19: oid 19 updating version 0 to 9 2026-02-25T08:41:28.317 INFO:tasks.rados.rados.0.trial039.stdout:19: oid 19 version 9 is already newer than 7 2026-02-25T08:41:28.317 INFO:tasks.rados.rados.0.trial039.stdout:19: oid 19 version 9 is already newer than 9 2026-02-25T08:41:28.317 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 19 v 9 (ObjNum 18 snap 0 seq_num 18) dirty exists 2026-02-25T08:41:28.317 INFO:tasks.rados.rados.0.trial039.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:41:28.317 INFO:tasks.rados.rados.0.trial039.stdout:19: done (15 left) 2026-02-25T08:41:28.317 INFO:tasks.rados.rados.0.trial039.stdout:20: write initial oid 20 2026-02-25T08:41:28.318 INFO:tasks.rados.rados.0.trial039.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2026-02-25T08:41:28.321 INFO:tasks.rados.rados.0.trial039.stdout:20: writing trial03942481-20 from 719333 to 1181770 tid 1 2026-02-25T08:41:28.325 INFO:tasks.rados.rados.0.trial039.stdout:20: writing trial03942481-20 from 1962099 to 2319333 tid 2 2026-02-25T08:41:28.325 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:28.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2026-02-25T08:41:28.672 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T08:41:28.672 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2026-02-25T08:41:28.672 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-25T08:41:28.672 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-25T08:41:28.675 INFO:tasks.ceph.osd.2:Started 2026-02-25T08:41:28.676 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:41:28.751 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:28.755 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:41:33.756 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:41:33.827 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:33.830 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:41:35.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-25T08:41:38.831 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:41:38.905 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:38.908 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:41:40.839 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~6s 2026-02-25T08:41:43.909 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:41:43.981 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:43.985 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:41:46.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2026-02-25T08:41:48.986 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:41:49.055 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:49.057 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:41:49.141 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:41:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:41:49.183 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:41:50.224 INFO:tasks.rados.rados.0.trial039.stdout:14: finishing write tid 1 to trial03942481-14 2026-02-25T08:41:50.230 INFO:tasks.rados.rados.0.trial039.stdout:11: finishing write tid 1 to trial03942481-11 2026-02-25T08:41:50.235 INFO:tasks.rados.rados.0.trial039.stdout:12: finishing write tid 1 to trial03942481-12 2026-02-25T08:41:50.247 INFO:tasks.rados.rados.0.trial039.stdout:14: finishing write tid 2 to trial03942481-14 2026-02-25T08:41:50.247 INFO:tasks.rados.rados.0.trial039.stdout:11: finishing write tid 2 to trial03942481-11 2026-02-25T08:41:50.251 INFO:tasks.rados.rados.0.trial039.stdout:11: finishing write tid 3 to trial03942481-11 2026-02-25T08:41:50.253 INFO:tasks.rados.rados.0.trial039.stdout:11: finishing write tid 4 to trial03942481-11 2026-02-25T08:41:50.253 INFO:tasks.rados.rados.0.trial039.stdout:11: oid 11 updating version 0 to 3 2026-02-25T08:41:50.253 INFO:tasks.rados.rados.0.trial039.stdout:11: oid 11 version 3 is already newer than 1 2026-02-25T08:41:50.253 INFO:tasks.rados.rados.0.trial039.stdout:11: oid 11 version 3 is already newer than 3 2026-02-25T08:41:50.253 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2026-02-25T08:41:50.253 INFO:tasks.rados.rados.0.trial039.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2026-02-25T08:41:50.253 INFO:tasks.rados.rados.0.trial039.stdout:11: done (15 left) 2026-02-25T08:41:50.253 INFO:tasks.rados.rados.0.trial039.stdout:21: write initial oid 21 2026-02-25T08:41:50.253 INFO:tasks.rados.rados.0.trial039.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2026-02-25T08:41:50.257 INFO:tasks.rados.rados.0.trial039.stdout:21: writing trial03942481-21 from 736140 to 1273826 tid 1 2026-02-25T08:41:50.262 INFO:tasks.rados.rados.0.trial039.stdout:21: writing trial03942481-21 from 2020581 to 2336140 tid 2 2026-02-25T08:41:50.262 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.262 INFO:tasks.rados.rados.0.trial039.stdout:12: finishing write tid 2 to trial03942481-12 2026-02-25T08:41:50.262 INFO:tasks.rados.rados.0.trial039.stdout:14: finishing write tid 3 to trial03942481-14 2026-02-25T08:41:50.262 INFO:tasks.rados.rados.0.trial039.stdout:12: finishing write tid 3 to trial03942481-12 2026-02-25T08:41:50.262 INFO:tasks.rados.rados.0.trial039.stdout:16: finishing write tid 1 to trial03942481-16 2026-02-25T08:41:50.263 INFO:tasks.rados.rados.0.trial039.stdout:12: finishing write tid 4 to trial03942481-12 2026-02-25T08:41:50.263 INFO:tasks.rados.rados.0.trial039.stdout:12: oid 12 updating version 0 to 3 2026-02-25T08:41:50.263 INFO:tasks.rados.rados.0.trial039.stdout:12: oid 12 version 3 is already newer than 1 2026-02-25T08:41:50.263 INFO:tasks.rados.rados.0.trial039.stdout:12: oid 12 version 3 is already newer than 3 2026-02-25T08:41:50.264 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2026-02-25T08:41:50.264 INFO:tasks.rados.rados.0.trial039.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2026-02-25T08:41:50.264 INFO:tasks.rados.rados.0.trial039.stdout:12: done (15 left) 2026-02-25T08:41:50.264 INFO:tasks.rados.rados.0.trial039.stdout:22: write initial oid 22 2026-02-25T08:41:50.264 INFO:tasks.rados.rados.0.trial039.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2026-02-25T08:41:50.268 INFO:tasks.rados.rados.0.trial039.stdout:22: writing trial03942481-22 from 752947 to 1365882 tid 1 2026-02-25T08:41:50.273 INFO:tasks.rados.rados.0.trial039.stdout:22: writing trial03942481-22 from 1962710 to 2352947 tid 2 2026-02-25T08:41:50.273 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.273 INFO:tasks.rados.rados.0.trial039.stdout:14: finishing write tid 4 to trial03942481-14 2026-02-25T08:41:50.274 INFO:tasks.rados.rados.0.trial039.stdout:14: oid 14 updating version 0 to 1 2026-02-25T08:41:50.274 INFO:tasks.rados.rados.0.trial039.stdout:14: oid 14 updating version 1 to 3 2026-02-25T08:41:50.274 INFO:tasks.rados.rados.0.trial039.stdout:14: oid 14 version 3 is already newer than 3 2026-02-25T08:41:50.274 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2026-02-25T08:41:50.274 INFO:tasks.rados.rados.0.trial039.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2026-02-25T08:41:50.274 INFO:tasks.rados.rados.0.trial039.stdout:14: done (15 left) 2026-02-25T08:41:50.274 INFO:tasks.rados.rados.0.trial039.stdout:23: write initial oid 23 2026-02-25T08:41:50.274 INFO:tasks.rados.rados.0.trial039.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2026-02-25T08:41:50.278 INFO:tasks.rados.rados.0.trial039.stdout:23: writing trial03942481-23 from 769754 to 1457938 tid 1 2026-02-25T08:41:50.282 INFO:tasks.rados.rados.0.trial039.stdout:23: writing trial03942481-23 from 2021192 to 2369754 tid 2 2026-02-25T08:41:50.282 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.282 INFO:tasks.rados.rados.0.trial039.stdout:20: finishing write tid 1 to trial03942481-20 2026-02-25T08:41:50.282 INFO:tasks.rados.rados.0.trial039.stdout:16: finishing write tid 2 to trial03942481-16 2026-02-25T08:41:50.282 INFO:tasks.rados.rados.0.trial039.stdout:2: finishing write tid 1 to trial03942481-2 2026-02-25T08:41:50.287 INFO:tasks.rados.rados.0.trial039.stdout:16: finishing write tid 3 to trial03942481-16 2026-02-25T08:41:50.290 INFO:tasks.rados.rados.0.trial039.stdout:16: finishing write tid 4 to trial03942481-16 2026-02-25T08:41:50.291 INFO:tasks.rados.rados.0.trial039.stdout:16: oid 16 updating version 0 to 4 2026-02-25T08:41:50.291 INFO:tasks.rados.rados.0.trial039.stdout:16: oid 16 updating version 4 to 6 2026-02-25T08:41:50.291 INFO:tasks.rados.rados.0.trial039.stdout:16: oid 16 version 6 is already newer than 6 2026-02-25T08:41:50.291 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2026-02-25T08:41:50.291 INFO:tasks.rados.rados.0.trial039.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2026-02-25T08:41:50.291 INFO:tasks.rados.rados.0.trial039.stdout:16: done (15 left) 2026-02-25T08:41:50.291 INFO:tasks.rados.rados.0.trial039.stdout:24: write initial oid 24 2026-02-25T08:41:50.291 INFO:tasks.rados.rados.0.trial039.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2026-02-25T08:41:50.295 INFO:tasks.rados.rados.0.trial039.stdout:24: writing trial03942481-24 from 786561 to 1266347 tid 1 2026-02-25T08:41:50.299 INFO:tasks.rados.rados.0.trial039.stdout:24: writing trial03942481-24 from 1796027 to 2263691 tid 2 2026-02-25T08:41:50.299 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.299 INFO:tasks.rados.rados.0.trial039.stdout:20: finishing write tid 2 to trial03942481-20 2026-02-25T08:41:50.300 INFO:tasks.rados.rados.0.trial039.stdout:8: finishing write tid 1 to trial03942481-8 2026-02-25T08:41:50.301 INFO:tasks.rados.rados.0.trial039.stdout:21: finishing write tid 1 to trial03942481-21 2026-02-25T08:41:50.303 INFO:tasks.rados.rados.0.trial039.stdout:9: finishing write tid 1 to trial03942481-9 2026-02-25T08:41:50.304 INFO:tasks.rados.rados.0.trial039.stdout:1: finishing write tid 1 to trial03942481-1 2026-02-25T08:41:50.305 INFO:tasks.rados.rados.0.trial039.stdout:20: finishing write tid 3 to trial03942481-20 2026-02-25T08:41:50.307 INFO:tasks.rados.rados.0.trial039.stdout:22: finishing write tid 1 to trial03942481-22 2026-02-25T08:41:50.311 INFO:tasks.rados.rados.0.trial039.stdout:20: finishing write tid 4 to trial03942481-20 2026-02-25T08:41:50.311 INFO:tasks.rados.rados.0.trial039.stdout:20: oid 20 updating version 0 to 6 2026-02-25T08:41:50.311 INFO:tasks.rados.rados.0.trial039.stdout:20: oid 20 version 6 is already newer than 6 2026-02-25T08:41:50.311 INFO:tasks.rados.rados.0.trial039.stdout:20: oid 20 version 6 is already newer than 4 2026-02-25T08:41:50.311 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2026-02-25T08:41:50.311 INFO:tasks.rados.rados.0.trial039.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:50.311 INFO:tasks.rados.rados.0.trial039.stdout:20: done (15 left) 2026-02-25T08:41:50.311 INFO:tasks.rados.rados.0.trial039.stdout:25: write initial oid 25 2026-02-25T08:41:50.311 INFO:tasks.rados.rados.0.trial039.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2026-02-25T08:41:50.314 INFO:tasks.rados.rados.0.trial039.stdout:25: writing trial03942481-25 from 403368 to 958403 tid 1 2026-02-25T08:41:50.319 INFO:tasks.rados.rados.0.trial039.stdout:25: writing trial03942481-25 from 1454509 to 2182184 tid 2 2026-02-25T08:41:50.320 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.330 INFO:tasks.rados.rados.0.trial039.stdout:4: finishing write tid 1 to trial03942481-4 2026-02-25T08:41:50.330 INFO:tasks.rados.rados.0.trial039.stdout:2: finishing write tid 2 to trial03942481-2 2026-02-25T08:41:50.332 INFO:tasks.rados.rados.0.trial039.stdout:24: finishing write tid 1 to trial03942481-24 2026-02-25T08:41:50.337 INFO:tasks.rados.rados.0.trial039.stdout:2: finishing write tid 3 to trial03942481-2 2026-02-25T08:41:50.345 INFO:tasks.rados.rados.0.trial039.stdout:7: finishing write tid 1 to trial03942481-7 2026-02-25T08:41:50.350 INFO:tasks.rados.rados.0.trial039.stdout:24: finishing write tid 2 to trial03942481-24 2026-02-25T08:41:50.353 INFO:tasks.rados.rados.0.trial039.stdout:24: finishing write tid 3 to trial03942481-24 2026-02-25T08:41:50.354 INFO:tasks.rados.rados.0.trial039.stdout:2: finishing write tid 4 to trial03942481-2 2026-02-25T08:41:50.354 INFO:tasks.rados.rados.0.trial039.stdout:2: oid 2 updating version 0 to 1 2026-02-25T08:41:50.354 INFO:tasks.rados.rados.0.trial039.stdout:2: oid 2 updating version 1 to 3 2026-02-25T08:41:50.354 INFO:tasks.rados.rados.0.trial039.stdout:2: oid 2 version 3 is already newer than 3 2026-02-25T08:41:50.354 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2026-02-25T08:41:50.354 INFO:tasks.rados.rados.0.trial039.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2026-02-25T08:41:50.354 INFO:tasks.rados.rados.0.trial039.stdout:2: done (15 left) 2026-02-25T08:41:50.354 INFO:tasks.rados.rados.0.trial039.stdout:26: write initial oid 26 2026-02-25T08:41:50.355 INFO:tasks.rados.rados.0.trial039.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2026-02-25T08:41:50.357 INFO:tasks.rados.rados.0.trial039.stdout:26: writing trial03942481-26 from 420175 to 1050459 tid 1 2026-02-25T08:41:50.363 INFO:tasks.rados.rados.0.trial039.stdout:26: writing trial03942481-26 from 1796638 to 2267971 tid 2 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:9: finishing write tid 2 to trial03942481-9 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:24: finishing write tid 4 to trial03942481-24 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:24: oid 24 updating version 0 to 3 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:24: oid 24 version 3 is already newer than 1 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:24: oid 24 version 3 is already newer than 3 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 24 v 3 (ObjNum 23 snap 0 seq_num 23) dirty exists 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:8: finishing write tid 2 to trial03942481-8 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:9: finishing write tid 3 to trial03942481-9 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:9: finishing write tid 4 to trial03942481-9 2026-02-25T08:41:50.364 INFO:tasks.rados.rados.0.trial039.stdout:9: oid 9 updating version 0 to 3 2026-02-25T08:41:50.365 INFO:tasks.rados.rados.0.trial039.stdout:9: oid 9 version 3 is already newer than 1 2026-02-25T08:41:50.365 INFO:tasks.rados.rados.0.trial039.stdout:9: oid 9 version 3 is already newer than 3 2026-02-25T08:41:50.365 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2026-02-25T08:41:50.365 INFO:tasks.rados.rados.0.trial039.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2026-02-25T08:41:50.365 INFO:tasks.rados.rados.0.trial039.stdout:25: finishing write tid 1 to trial03942481-25 2026-02-25T08:41:50.365 INFO:tasks.rados.rados.0.trial039.stdout:9: done (15 left) 2026-02-25T08:41:50.365 INFO:tasks.rados.rados.0.trial039.stdout:24: done (14 left) 2026-02-25T08:41:50.365 INFO:tasks.rados.rados.0.trial039.stdout:27: write initial oid 27 2026-02-25T08:41:50.365 INFO:tasks.rados.rados.0.trial039.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2026-02-25T08:41:50.368 INFO:tasks.rados.rados.0.trial039.stdout:27: writing trial03942481-27 from 436982 to 1142515 tid 1 2026-02-25T08:41:50.373 INFO:tasks.rados.rados.0.trial039.stdout:27: writing trial03942481-27 from 1855120 to 2436982 tid 2 2026-02-25T08:41:50.374 INFO:tasks.rados.rados.0.trial039.stdout:28: write initial oid 28 2026-02-25T08:41:50.374 INFO:tasks.rados.rados.0.trial039.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2026-02-25T08:41:50.378 INFO:tasks.rados.rados.0.trial039.stdout:28: writing trial03942481-28 from 453789 to 1234571 tid 1 2026-02-25T08:41:50.383 INFO:tasks.rados.rados.0.trial039.stdout:28: writing trial03942481-28 from 1913602 to 2388604 tid 2 2026-02-25T08:41:50.383 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.383 INFO:tasks.rados.rados.0.trial039.stdout:8: finishing write tid 3 to trial03942481-8 2026-02-25T08:41:50.383 INFO:tasks.rados.rados.0.trial039.stdout:4: finishing write tid 2 to trial03942481-4 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:22: finishing write tid 2 to trial03942481-22 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:1: finishing write tid 2 to trial03942481-1 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:1: finishing write tid 3 to trial03942481-1 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:21: finishing write tid 2 to trial03942481-21 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:8: finishing write tid 4 to trial03942481-8 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:8: oid 8 updating version 0 to 3 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:8: oid 8 version 3 is already newer than 3 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:8: oid 8 version 3 is already newer than 1 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:4: finishing write tid 3 to trial03942481-4 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:4: finishing write tid 4 to trial03942481-4 2026-02-25T08:41:50.384 INFO:tasks.rados.rados.0.trial039.stdout:4: oid 4 updating version 0 to 1 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:4: oid 4 updating version 1 to 3 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:4: oid 4 version 3 is already newer than 3 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:1: finishing write tid 4 to trial03942481-1 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:1: oid 1 updating version 0 to 1 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:1: oid 1 updating version 1 to 3 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:1: oid 1 version 3 is already newer than 3 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:13: finishing write tid 1 to trial03942481-13 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:1: done (15 left) 2026-02-25T08:41:50.385 INFO:tasks.rados.rados.0.trial039.stdout:4: done (14 left) 2026-02-25T08:41:50.386 INFO:tasks.rados.rados.0.trial039.stdout:8: done (13 left) 2026-02-25T08:41:50.386 INFO:tasks.rados.rados.0.trial039.stdout:29: write initial oid 29 2026-02-25T08:41:50.386 INFO:tasks.rados.rados.0.trial039.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2026-02-25T08:41:50.386 INFO:tasks.rados.rados.0.trial039.stdout:29: writing trial03942481-29 from 470596 to 926627 tid 1 2026-02-25T08:41:50.392 INFO:tasks.rados.rados.0.trial039.stdout:29: writing trial03942481-29 from 1572084 to 2190744 tid 2 2026-02-25T08:41:50.392 INFO:tasks.rados.rados.0.trial039.stdout:30: write initial oid 30 2026-02-25T08:41:50.392 INFO:tasks.rados.rados.0.trial039.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2026-02-25T08:41:50.395 INFO:tasks.rados.rados.0.trial039.stdout:30: writing trial03942481-30 from 487403 to 1018683 tid 1 2026-02-25T08:41:50.399 INFO:tasks.rados.rados.0.trial039.stdout:30: writing trial03942481-30 from 1514213 to 1992884 tid 2 2026-02-25T08:41:50.399 INFO:tasks.rados.rados.0.trial039.stdout:31: write initial oid 31 2026-02-25T08:41:50.399 INFO:tasks.rados.rados.0.trial039.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2026-02-25T08:41:50.403 INFO:tasks.rados.rados.0.trial039.stdout:31: writing trial03942481-31 from 504210 to 1110739 tid 1 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout:31: writing trial03942481-31 from 1572695 to 2195024 tid 2 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout:22: finishing write tid 3 to trial03942481-22 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout:22: finishing write tid 4 to trial03942481-22 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout:22: oid 22 updating version 0 to 1 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout:22: oid 22 updating version 1 to 3 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout:22: oid 22 version 3 is already newer than 3 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout:21: finishing write tid 3 to trial03942481-21 2026-02-25T08:41:50.408 INFO:tasks.rados.rados.0.trial039.stdout:21: finishing write tid 4 to trial03942481-21 2026-02-25T08:41:50.409 INFO:tasks.rados.rados.0.trial039.stdout:21: oid 21 updating version 0 to 12 2026-02-25T08:41:50.409 INFO:tasks.rados.rados.0.trial039.stdout:21: oid 21 version 12 is already newer than 10 2026-02-25T08:41:50.409 INFO:tasks.rados.rados.0.trial039.stdout:21: oid 21 version 12 is already newer than 12 2026-02-25T08:41:50.409 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 21 v 12 (ObjNum 20 snap 0 seq_num 20) dirty exists 2026-02-25T08:41:50.409 INFO:tasks.rados.rados.0.trial039.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2026-02-25T08:41:50.409 INFO:tasks.rados.rados.0.trial039.stdout:26: finishing write tid 1 to trial03942481-26 2026-02-25T08:41:50.409 INFO:tasks.rados.rados.0.trial039.stdout:21: done (15 left) 2026-02-25T08:41:50.409 INFO:tasks.rados.rados.0.trial039.stdout:22: done (14 left) 2026-02-25T08:41:50.409 INFO:tasks.rados.rados.0.trial039.stdout:32: write initial oid 32 2026-02-25T08:41:50.409 INFO:tasks.rados.rados.0.trial039.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2026-02-25T08:41:50.412 INFO:tasks.rados.rados.0.trial039.stdout:32: writing trial03942481-32 from 521017 to 1319148 tid 1 2026-02-25T08:41:50.416 INFO:tasks.rados.rados.0.trial039.stdout:32: writing trial03942481-32 from 1747530 to 2229870 tid 2 2026-02-25T08:41:50.416 INFO:tasks.rados.rados.0.trial039.stdout:33: write initial oid 33 2026-02-25T08:41:50.416 INFO:tasks.rados.rados.0.trial039.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2026-02-25T08:41:50.419 INFO:tasks.rados.rados.0.trial039.stdout:33: writing trial03942481-33 from 537824 to 1011204 tid 1 2026-02-25T08:41:50.426 INFO:tasks.rados.rados.0.trial039.stdout:33: writing trial03942481-33 from 1806012 to 2432010 tid 2 2026-02-25T08:41:50.426 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.426 INFO:tasks.rados.rados.0.trial039.stdout:25: finishing write tid 2 to trial03942481-25 2026-02-25T08:41:50.426 INFO:tasks.rados.rados.0.trial039.stdout:23: finishing write tid 1 to trial03942481-23 2026-02-25T08:41:50.426 INFO:tasks.rados.rados.0.trial039.stdout:27: finishing write tid 1 to trial03942481-27 2026-02-25T08:41:50.426 INFO:tasks.rados.rados.0.trial039.stdout:7: finishing write tid 2 to trial03942481-7 2026-02-25T08:41:50.426 INFO:tasks.rados.rados.0.trial039.stdout:25: finishing write tid 3 to trial03942481-25 2026-02-25T08:41:50.426 INFO:tasks.rados.rados.0.trial039.stdout:25: finishing write tid 4 to trial03942481-25 2026-02-25T08:41:50.427 INFO:tasks.rados.rados.0.trial039.stdout:25: oid 25 updating version 0 to 3 2026-02-25T08:41:50.427 INFO:tasks.rados.rados.0.trial039.stdout:25: oid 25 version 3 is already newer than 1 2026-02-25T08:41:50.427 INFO:tasks.rados.rados.0.trial039.stdout:25: oid 25 version 3 is already newer than 3 2026-02-25T08:41:50.427 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 25 v 3 (ObjNum 24 snap 0 seq_num 24) dirty exists 2026-02-25T08:41:50.427 INFO:tasks.rados.rados.0.trial039.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2026-02-25T08:41:50.427 INFO:tasks.rados.rados.0.trial039.stdout:10: finishing write tid 1 to trial03942481-10 2026-02-25T08:41:50.427 INFO:tasks.rados.rados.0.trial039.stdout:26: finishing write tid 2 to trial03942481-26 2026-02-25T08:41:50.427 INFO:tasks.rados.rados.0.trial039.stdout:25: done (15 left) 2026-02-25T08:41:50.427 INFO:tasks.rados.rados.0.trial039.stdout:34: write initial oid 34 2026-02-25T08:41:50.427 INFO:tasks.rados.rados.0.trial039.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2026-02-25T08:41:50.429 INFO:tasks.rados.rados.0.trial039.stdout:34: writing trial03942481-34 from 554631 to 1103260 tid 1 2026-02-25T08:41:50.434 INFO:tasks.rados.rados.0.trial039.stdout:34: writing trial03942481-34 from 1748141 to 2234150 tid 2 2026-02-25T08:41:50.434 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.434 INFO:tasks.rados.rados.0.trial039.stdout:13: finishing write tid 2 to trial03942481-13 2026-02-25T08:41:50.440 INFO:tasks.rados.rados.0.trial039.stdout:7: finishing write tid 3 to trial03942481-7 2026-02-25T08:41:50.445 INFO:tasks.rados.rados.0.trial039.stdout:13: finishing write tid 3 to trial03942481-13 2026-02-25T08:41:50.447 INFO:tasks.rados.rados.0.trial039.stdout:13: finishing write tid 4 to trial03942481-13 2026-02-25T08:41:50.447 INFO:tasks.rados.rados.0.trial039.stdout:13: oid 13 updating version 0 to 6 2026-02-25T08:41:50.447 INFO:tasks.rados.rados.0.trial039.stdout:13: oid 13 version 6 is already newer than 4 2026-02-25T08:41:50.447 INFO:tasks.rados.rados.0.trial039.stdout:13: oid 13 version 6 is already newer than 6 2026-02-25T08:41:50.447 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2026-02-25T08:41:50.447 INFO:tasks.rados.rados.0.trial039.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2026-02-25T08:41:50.447 INFO:tasks.rados.rados.0.trial039.stdout:13: done (15 left) 2026-02-25T08:41:50.447 INFO:tasks.rados.rados.0.trial039.stdout:35: write initial oid 35 2026-02-25T08:41:50.447 INFO:tasks.rados.rados.0.trial039.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2026-02-25T08:41:50.451 INFO:tasks.rados.rados.0.trial039.stdout:35: writing trial03942481-35 from 571438 to 1195316 tid 1 2026-02-25T08:41:50.457 INFO:tasks.rados.rados.0.trial039.stdout:35: writing trial03942481-35 from 1806623 to 2436290 tid 2 2026-02-25T08:41:50.457 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.457 INFO:tasks.rados.rados.0.trial039.stdout:7: finishing write tid 4 to trial03942481-7 2026-02-25T08:41:50.457 INFO:tasks.rados.rados.0.trial039.stdout:7: oid 7 updating version 0 to 4 2026-02-25T08:41:50.458 INFO:tasks.rados.rados.0.trial039.stdout:7: oid 7 updating version 4 to 6 2026-02-25T08:41:50.458 INFO:tasks.rados.rados.0.trial039.stdout:7: oid 7 version 6 is already newer than 6 2026-02-25T08:41:50.458 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 7 v 6 (ObjNum 6 snap 0 seq_num 6) dirty exists 2026-02-25T08:41:50.458 INFO:tasks.rados.rados.0.trial039.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:41:50.458 INFO:tasks.rados.rados.0.trial039.stdout:23: finishing write tid 2 to trial03942481-23 2026-02-25T08:41:50.458 INFO:tasks.rados.rados.0.trial039.stdout:15: finishing write tid 1 to trial03942481-15 2026-02-25T08:41:50.458 INFO:tasks.rados.rados.0.trial039.stdout:7: done (15 left) 2026-02-25T08:41:50.458 INFO:tasks.rados.rados.0.trial039.stdout:36: write initial oid 36 2026-02-25T08:41:50.458 INFO:tasks.rados.rados.0.trial039.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2026-02-25T08:41:50.461 INFO:tasks.rados.rados.0.trial039.stdout:36: writing trial03942481-36 from 588245 to 1287372 tid 1 2026-02-25T08:41:50.466 INFO:tasks.rados.rados.0.trial039.stdout:36: writing trial03942481-36 from 1865105 to 2354783 tid 2 2026-02-25T08:41:50.466 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.466 INFO:tasks.rados.rados.0.trial039.stdout:23: finishing write tid 3 to trial03942481-23 2026-02-25T08:41:50.466 INFO:tasks.rados.rados.0.trial039.stdout:23: finishing write tid 4 to trial03942481-23 2026-02-25T08:41:50.466 INFO:tasks.rados.rados.0.trial039.stdout:23: oid 23 updating version 0 to 4 2026-02-25T08:41:50.466 INFO:tasks.rados.rados.0.trial039.stdout:23: oid 23 updating version 4 to 6 2026-02-25T08:41:50.466 INFO:tasks.rados.rados.0.trial039.stdout:23: oid 23 version 6 is already newer than 6 2026-02-25T08:41:50.466 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2026-02-25T08:41:50.466 INFO:tasks.rados.rados.0.trial039.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2026-02-25T08:41:50.466 INFO:tasks.rados.rados.0.trial039.stdout:23: done (15 left) 2026-02-25T08:41:50.467 INFO:tasks.rados.rados.0.trial039.stdout:37: write initial oid 37 2026-02-25T08:41:50.467 INFO:tasks.rados.rados.0.trial039.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2026-02-25T08:41:50.471 INFO:tasks.rados.rados.0.trial039.stdout:37: writing trial03942481-37 from 605052 to 1379428 tid 1 2026-02-25T08:41:50.476 INFO:tasks.rados.rados.0.trial039.stdout:37: writing trial03942481-37 from 1923587 to 2556923 tid 2 2026-02-25T08:41:50.476 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.476 INFO:tasks.rados.rados.0.trial039.stdout:27: finishing write tid 2 to trial03942481-27 2026-02-25T08:41:50.476 INFO:tasks.rados.rados.0.trial039.stdout:15: finishing write tid 2 to trial03942481-15 2026-02-25T08:41:50.476 INFO:tasks.rados.rados.0.trial039.stdout:27: finishing write tid 3 to trial03942481-27 2026-02-25T08:41:50.476 INFO:tasks.rados.rados.0.trial039.stdout:10: finishing write tid 2 to trial03942481-10 2026-02-25T08:41:50.477 INFO:tasks.rados.rados.0.trial039.stdout:15: finishing write tid 3 to trial03942481-15 2026-02-25T08:41:50.481 INFO:tasks.rados.rados.0.trial039.stdout:27: finishing write tid 4 to trial03942481-27 2026-02-25T08:41:50.481 INFO:tasks.rados.rados.0.trial039.stdout:27: oid 27 updating version 0 to 1 2026-02-25T08:41:50.481 INFO:tasks.rados.rados.0.trial039.stdout:27: oid 27 updating version 1 to 3 2026-02-25T08:41:50.481 INFO:tasks.rados.rados.0.trial039.stdout:27: oid 27 version 3 is already newer than 3 2026-02-25T08:41:50.481 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 27 v 3 (ObjNum 26 snap 0 seq_num 26) dirty exists 2026-02-25T08:41:50.481 INFO:tasks.rados.rados.0.trial039.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2026-02-25T08:41:50.482 INFO:tasks.rados.rados.0.trial039.stdout:27: done (15 left) 2026-02-25T08:41:50.482 INFO:tasks.rados.rados.0.trial039.stdout:38: write initial oid 38 2026-02-25T08:41:50.482 INFO:tasks.rados.rados.0.trial039.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2026-02-25T08:41:50.485 INFO:tasks.rados.rados.0.trial039.stdout:38: writing trial03942481-38 from 621859 to 1071484 tid 1 2026-02-25T08:41:50.492 INFO:tasks.rados.rados.0.trial039.stdout:38: writing trial03942481-38 from 1865716 to 2621859 tid 2 2026-02-25T08:41:50.492 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:18: finishing write tid 1 to trial03942481-18 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:26: finishing write tid 3 to trial03942481-26 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:10: finishing write tid 3 to trial03942481-10 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:15: finishing write tid 4 to trial03942481-15 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:15: oid 15 updating version 0 to 9 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:15: oid 15 version 9 is already newer than 7 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:15: oid 15 version 9 is already newer than 9 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:26: finishing write tid 4 to trial03942481-26 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:26: oid 26 updating version 0 to 6 2026-02-25T08:41:50.493 INFO:tasks.rados.rados.0.trial039.stdout:26: oid 26 version 6 is already newer than 6 2026-02-25T08:41:50.494 INFO:tasks.rados.rados.0.trial039.stdout:26: oid 26 version 6 is already newer than 4 2026-02-25T08:41:50.494 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2026-02-25T08:41:50.494 INFO:tasks.rados.rados.0.trial039.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2026-02-25T08:41:50.494 INFO:tasks.rados.rados.0.trial039.stdout:30: finishing write tid 1 to trial03942481-30 2026-02-25T08:41:50.494 INFO:tasks.rados.rados.0.trial039.stdout:15: done (15 left) 2026-02-25T08:41:50.494 INFO:tasks.rados.rados.0.trial039.stdout:26: done (14 left) 2026-02-25T08:41:50.494 INFO:tasks.rados.rados.0.trial039.stdout:39: write initial oid 39 2026-02-25T08:41:50.494 INFO:tasks.rados.rados.0.trial039.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2026-02-25T08:41:50.496 INFO:tasks.rados.rados.0.trial039.stdout:39: writing trial03942481-39 from 638666 to 1163540 tid 1 2026-02-25T08:41:50.501 INFO:tasks.rados.rados.0.trial039.stdout:39: writing trial03942481-39 from 1924198 to 2561203 tid 2 2026-02-25T08:41:50.501 INFO:tasks.rados.rados.0.trial039.stdout:40: write initial oid 40 2026-02-25T08:41:50.501 INFO:tasks.rados.rados.0.trial039.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2026-02-25T08:41:50.506 INFO:tasks.rados.rados.0.trial039.stdout:40: writing trial03942481-40 from 655473 to 1371949 tid 1 2026-02-25T08:41:50.512 INFO:tasks.rados.rados.0.trial039.stdout:40: writing trial03942481-40 from 2099033 to 2655473 tid 2 2026-02-25T08:41:50.512 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.512 INFO:tasks.rados.rados.0.trial039.stdout:10: finishing write tid 4 to trial03942481-10 2026-02-25T08:41:50.512 INFO:tasks.rados.rados.0.trial039.stdout:10: oid 10 updating version 0 to 6 2026-02-25T08:41:50.512 INFO:tasks.rados.rados.0.trial039.stdout:10: oid 10 version 6 is already newer than 6 2026-02-25T08:41:50.512 INFO:tasks.rados.rados.0.trial039.stdout:10: oid 10 version 6 is already newer than 4 2026-02-25T08:41:50.512 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2026-02-25T08:41:50.512 INFO:tasks.rados.rados.0.trial039.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2026-02-25T08:41:50.512 INFO:tasks.rados.rados.0.trial039.stdout:35: finishing write tid 1 to trial03942481-35 2026-02-25T08:41:50.513 INFO:tasks.rados.rados.0.trial039.stdout:17: finishing write tid 1 to trial03942481-17 2026-02-25T08:41:50.513 INFO:tasks.rados.rados.0.trial039.stdout:18: finishing write tid 2 to trial03942481-18 2026-02-25T08:41:50.513 INFO:tasks.rados.rados.0.trial039.stdout:29: finishing write tid 1 to trial03942481-29 2026-02-25T08:41:50.513 INFO:tasks.rados.rados.0.trial039.stdout:10: done (15 left) 2026-02-25T08:41:50.513 INFO:tasks.rados.rados.0.trial039.stdout:41: write initial oid 41 2026-02-25T08:41:50.513 INFO:tasks.rados.rados.0.trial039.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2026-02-25T08:41:50.517 INFO:tasks.rados.rados.0.trial039.stdout:41: writing trial03942481-41 from 672280 to 1464005 tid 1 2026-02-25T08:41:50.522 INFO:tasks.rados.rados.0.trial039.stdout:41: writing trial03942481-41 from 2157515 to 2672280 tid 2 2026-02-25T08:41:50.522 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.522 INFO:tasks.rados.rados.0.trial039.stdout:28: finishing write tid 1 to trial03942481-28 2026-02-25T08:41:50.522 INFO:tasks.rados.rados.0.trial039.stdout:18: finishing write tid 3 to trial03942481-18 2026-02-25T08:41:50.523 INFO:tasks.rados.rados.0.trial039.stdout:18: finishing write tid 4 to trial03942481-18 2026-02-25T08:41:50.523 INFO:tasks.rados.rados.0.trial039.stdout:18: oid 18 updating version 0 to 12 2026-02-25T08:41:50.523 INFO:tasks.rados.rados.0.trial039.stdout:18: oid 18 version 12 is already newer than 10 2026-02-25T08:41:50.523 INFO:tasks.rados.rados.0.trial039.stdout:18: oid 18 version 12 is already newer than 12 2026-02-25T08:41:50.523 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 18 v 12 (ObjNum 17 snap 0 seq_num 17) dirty exists 2026-02-25T08:41:50.524 INFO:tasks.rados.rados.0.trial039.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2026-02-25T08:41:50.524 INFO:tasks.rados.rados.0.trial039.stdout:18: done (15 left) 2026-02-25T08:41:50.524 INFO:tasks.rados.rados.0.trial039.stdout:42: write initial oid 42 2026-02-25T08:41:50.524 INFO:tasks.rados.rados.0.trial039.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2026-02-25T08:41:50.527 INFO:tasks.rados.rados.0.trial039.stdout:42: writing trial03942481-42 from 689087 to 1156061 tid 1 2026-02-25T08:41:50.532 INFO:tasks.rados.rados.0.trial039.stdout:42: writing trial03942481-42 from 1699644 to 2483976 tid 2 2026-02-25T08:41:50.533 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.533 INFO:tasks.rados.rados.0.trial039.stdout:38: finishing write tid 1 to trial03942481-38 2026-02-25T08:41:50.539 INFO:tasks.rados.rados.0.trial039.stdout:35: finishing write tid 2 to trial03942481-35 2026-02-25T08:41:50.552 INFO:tasks.rados.rados.0.trial039.stdout:38: finishing write tid 2 to trial03942481-38 2026-02-25T08:41:50.553 INFO:tasks.rados.rados.0.trial039.stdout:30: finishing write tid 2 to trial03942481-30 2026-02-25T08:41:50.553 INFO:tasks.rados.rados.0.trial039.stdout:41: finishing write tid 1 to trial03942481-41 2026-02-25T08:41:50.558 INFO:tasks.rados.rados.0.trial039.stdout:35: finishing write tid 3 to trial03942481-35 2026-02-25T08:41:50.559 INFO:tasks.rados.rados.0.trial039.stdout:38: finishing write tid 3 to trial03942481-38 2026-02-25T08:41:50.560 INFO:tasks.rados.rados.0.trial039.stdout:35: finishing write tid 4 to trial03942481-35 2026-02-25T08:41:50.560 INFO:tasks.rados.rados.0.trial039.stdout:35: oid 35 updating version 0 to 1 2026-02-25T08:41:50.560 INFO:tasks.rados.rados.0.trial039.stdout:35: oid 35 updating version 1 to 3 2026-02-25T08:41:50.560 INFO:tasks.rados.rados.0.trial039.stdout:35: oid 35 version 3 is already newer than 3 2026-02-25T08:41:50.560 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 35 v 3 (ObjNum 34 snap 0 seq_num 34) dirty exists 2026-02-25T08:41:50.560 INFO:tasks.rados.rados.0.trial039.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2026-02-25T08:41:50.561 INFO:tasks.rados.rados.0.trial039.stdout:35: done (15 left) 2026-02-25T08:41:50.561 INFO:tasks.rados.rados.0.trial039.stdout:43: write initial oid 43 2026-02-25T08:41:50.561 INFO:tasks.rados.rados.0.trial039.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2026-02-25T08:41:50.564 INFO:tasks.rados.rados.0.trial039.stdout:43: writing trial03942481-43 from 705894 to 1248117 tid 1 2026-02-25T08:41:50.570 INFO:tasks.rados.rados.0.trial039.stdout:43: writing trial03942481-43 from 1758126 to 2402469 tid 2 2026-02-25T08:41:50.570 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.570 INFO:tasks.rados.rados.0.trial039.stdout:38: finishing write tid 4 to trial03942481-38 2026-02-25T08:41:50.570 INFO:tasks.rados.rados.0.trial039.stdout:38: oid 38 updating version 0 to 15 2026-02-25T08:41:50.570 INFO:tasks.rados.rados.0.trial039.stdout:38: oid 38 version 15 is already newer than 15 2026-02-25T08:41:50.570 INFO:tasks.rados.rados.0.trial039.stdout:38: oid 38 version 15 is already newer than 13 2026-02-25T08:41:50.570 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 38 v 15 (ObjNum 37 snap 0 seq_num 37) dirty exists 2026-02-25T08:41:50.570 INFO:tasks.rados.rados.0.trial039.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:41:50.570 INFO:tasks.rados.rados.0.trial039.stdout:28: finishing write tid 2 to trial03942481-28 2026-02-25T08:41:50.570 INFO:tasks.rados.rados.0.trial039.stdout:38: done (15 left) 2026-02-25T08:41:50.571 INFO:tasks.rados.rados.0.trial039.stdout:44: write initial oid 44 2026-02-25T08:41:50.571 INFO:tasks.rados.rados.0.trial039.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2026-02-25T08:41:50.574 INFO:tasks.rados.rados.0.trial039.stdout:44: writing trial03942481-44 from 722701 to 1340173 tid 1 2026-02-25T08:41:50.580 INFO:tasks.rados.rados.0.trial039.stdout:44: writing trial03942481-44 from 1816608 to 2604609 tid 2 2026-02-25T08:41:50.580 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.580 INFO:tasks.rados.rados.0.trial039.stdout:28: finishing write tid 3 to trial03942481-28 2026-02-25T08:41:50.580 INFO:tasks.rados.rados.0.trial039.stdout:30: finishing write tid 3 to trial03942481-30 2026-02-25T08:41:50.580 INFO:tasks.rados.rados.0.trial039.stdout:28: finishing write tid 4 to trial03942481-28 2026-02-25T08:41:50.580 INFO:tasks.rados.rados.0.trial039.stdout:28: oid 28 updating version 0 to 7 2026-02-25T08:41:50.580 INFO:tasks.rados.rados.0.trial039.stdout:28: oid 28 updating version 7 to 9 2026-02-25T08:41:50.580 INFO:tasks.rados.rados.0.trial039.stdout:28: oid 28 version 9 is already newer than 9 2026-02-25T08:41:50.581 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2026-02-25T08:41:50.581 INFO:tasks.rados.rados.0.trial039.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2026-02-25T08:41:50.581 INFO:tasks.rados.rados.0.trial039.stdout:17: finishing write tid 2 to trial03942481-17 2026-02-25T08:41:50.581 INFO:tasks.rados.rados.0.trial039.stdout:28: done (15 left) 2026-02-25T08:41:50.581 INFO:tasks.rados.rados.0.trial039.stdout:45: write initial oid 45 2026-02-25T08:41:50.581 INFO:tasks.rados.rados.0.trial039.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2026-02-25T08:41:50.585 INFO:tasks.rados.rados.0.trial039.stdout:45: writing trial03942481-45 from 739508 to 1432229 tid 1 2026-02-25T08:41:50.589 INFO:tasks.rados.rados.0.trial039.stdout:45: writing trial03942481-45 from 1875090 to 2523102 tid 2 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout:30: finishing write tid 4 to trial03942481-30 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout:30: oid 30 updating version 0 to 6 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout:30: oid 30 version 6 is already newer than 4 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout:30: oid 30 version 6 is already newer than 6 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout:33: finishing write tid 1 to trial03942481-33 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout:30: done (15 left) 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout:46: write initial oid 46 2026-02-25T08:41:50.590 INFO:tasks.rados.rados.0.trial039.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2026-02-25T08:41:50.595 INFO:tasks.rados.rados.0.trial039.stdout:46: writing trial03942481-46 from 756315 to 1524285 tid 1 2026-02-25T08:41:50.600 INFO:tasks.rados.rados.0.trial039.stdout:46: writing trial03942481-46 from 1933572 to 2725242 tid 2 2026-02-25T08:41:50.600 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.600 INFO:tasks.rados.rados.0.trial039.stdout:29: finishing write tid 2 to trial03942481-29 2026-02-25T08:41:50.600 INFO:tasks.rados.rados.0.trial039.stdout:17: finishing write tid 3 to trial03942481-17 2026-02-25T08:41:50.601 INFO:tasks.rados.rados.0.trial039.stdout:33: finishing write tid 2 to trial03942481-33 2026-02-25T08:41:50.609 INFO:tasks.rados.rados.0.trial039.stdout:17: finishing write tid 4 to trial03942481-17 2026-02-25T08:41:50.609 INFO:tasks.rados.rados.0.trial039.stdout:17: oid 17 updating version 0 to 7 2026-02-25T08:41:50.610 INFO:tasks.rados.rados.0.trial039.stdout:17: oid 17 updating version 7 to 9 2026-02-25T08:41:50.610 INFO:tasks.rados.rados.0.trial039.stdout:17: oid 17 version 9 is already newer than 9 2026-02-25T08:41:50.610 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2026-02-25T08:41:50.610 INFO:tasks.rados.rados.0.trial039.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2026-02-25T08:41:50.610 INFO:tasks.rados.rados.0.trial039.stdout:17: done (15 left) 2026-02-25T08:41:50.610 INFO:tasks.rados.rados.0.trial039.stdout:47: write initial oid 47 2026-02-25T08:41:50.610 INFO:tasks.rados.rados.0.trial039.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2026-02-25T08:41:50.615 INFO:tasks.rados.rados.0.trial039.stdout:47: writing trial03942481-47 from 773122 to 1332694 tid 1 2026-02-25T08:41:50.620 INFO:tasks.rados.rados.0.trial039.stdout:47: writing trial03942481-47 from 1992054 to 2643735 tid 2 2026-02-25T08:41:50.620 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.620 INFO:tasks.rados.rados.0.trial039.stdout:29: finishing write tid 3 to trial03942481-29 2026-02-25T08:41:50.620 INFO:tasks.rados.rados.0.trial039.stdout:33: finishing write tid 3 to trial03942481-33 2026-02-25T08:41:50.621 INFO:tasks.rados.rados.0.trial039.stdout:29: finishing write tid 4 to trial03942481-29 2026-02-25T08:41:50.621 INFO:tasks.rados.rados.0.trial039.stdout:29: oid 29 updating version 0 to 9 2026-02-25T08:41:50.621 INFO:tasks.rados.rados.0.trial039.stdout:29: oid 29 version 9 is already newer than 9 2026-02-25T08:41:50.621 INFO:tasks.rados.rados.0.trial039.stdout:29: oid 29 version 9 is already newer than 7 2026-02-25T08:41:50.621 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2026-02-25T08:41:50.622 INFO:tasks.rados.rados.0.trial039.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2026-02-25T08:41:50.622 INFO:tasks.rados.rados.0.trial039.stdout:29: done (15 left) 2026-02-25T08:41:50.622 INFO:tasks.rados.rados.0.trial039.stdout:48: write initial oid 48 2026-02-25T08:41:50.622 INFO:tasks.rados.rados.0.trial039.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2026-02-25T08:41:50.626 INFO:tasks.rados.rados.0.trial039.stdout:48: writing trial03942481-48 from 789929 to 1424750 tid 1 2026-02-25T08:41:50.631 INFO:tasks.rados.rados.0.trial039.stdout:48: writing trial03942481-48 from 2050536 to 2789929 tid 2 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout:33: finishing write tid 4 to trial03942481-33 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout:33: oid 33 updating version 0 to 12 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout:33: oid 33 version 12 is already newer than 10 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout:33: oid 33 version 12 is already newer than 12 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 33 v 12 (ObjNum 32 snap 0 seq_num 32) dirty exists 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout:46: finishing write tid 1 to trial03942481-46 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout:33: done (15 left) 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout:49: write initial oid 49 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial039.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2026-02-25T08:41:50.635 INFO:tasks.rados.rados.0.trial039.stdout:49: writing trial03942481-49 from 406736 to 1116806 tid 1 2026-02-25T08:41:50.640 INFO:tasks.rados.rados.0.trial039.stdout:49: writing trial03942481-49 from 1709018 to 2364368 tid 2 2026-02-25T08:41:50.640 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.640 INFO:tasks.rados.rados.0.trial039.stdout:32: finishing write tid 1 to trial03942481-32 2026-02-25T08:41:50.640 INFO:tasks.rados.rados.0.trial039.stdout:31: finishing write tid 1 to trial03942481-31 2026-02-25T08:41:50.642 INFO:tasks.rados.rados.0.trial039.stdout:43: finishing write tid 1 to trial03942481-43 2026-02-25T08:41:50.660 INFO:tasks.rados.rados.0.trial039.stdout:34: finishing write tid 1 to trial03942481-34 2026-02-25T08:41:50.661 INFO:tasks.rados.rados.0.trial039.stdout:37: finishing write tid 1 to trial03942481-37 2026-02-25T08:41:50.662 INFO:tasks.rados.rados.0.trial039.stdout:47: finishing write tid 1 to trial03942481-47 2026-02-25T08:41:50.682 INFO:tasks.rados.rados.0.trial039.stdout:41: finishing write tid 2 to trial03942481-41 2026-02-25T08:41:50.684 INFO:tasks.rados.rados.0.trial039.stdout:48: finishing write tid 1 to trial03942481-48 2026-02-25T08:41:50.694 INFO:tasks.rados.rados.0.trial039.stdout:41: finishing write tid 3 to trial03942481-41 2026-02-25T08:41:50.696 INFO:tasks.rados.rados.0.trial039.stdout:41: finishing write tid 4 to trial03942481-41 2026-02-25T08:41:50.696 INFO:tasks.rados.rados.0.trial039.stdout:41: oid 41 updating version 0 to 6 2026-02-25T08:41:50.696 INFO:tasks.rados.rados.0.trial039.stdout:41: oid 41 version 6 is already newer than 6 2026-02-25T08:41:50.696 INFO:tasks.rados.rados.0.trial039.stdout:41: oid 41 version 6 is already newer than 4 2026-02-25T08:41:50.696 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2026-02-25T08:41:50.697 INFO:tasks.rados.rados.0.trial039.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2026-02-25T08:41:50.697 INFO:tasks.rados.rados.0.trial039.stdout:41: done (15 left) 2026-02-25T08:41:50.697 INFO:tasks.rados.rados.0.trial039.stdout:50: write initial oid 50 2026-02-25T08:41:50.697 INFO:tasks.rados.rados.0.trial039.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial039.stdout:50: writing trial03942481-50 from 423543 to 1208862 tid 1 2026-02-25T08:41:50.706 INFO:tasks.rados.rados.0.trial039.stdout:50: writing trial03942481-50 from 1767500 to 2566508 tid 2 2026-02-25T08:41:50.706 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.706 INFO:tasks.rados.rados.0.trial039.stdout:34: finishing write tid 2 to trial03942481-34 2026-02-25T08:41:50.711 INFO:tasks.rados.rados.0.trial039.stdout:43: finishing write tid 2 to trial03942481-43 2026-02-25T08:41:50.715 INFO:tasks.rados.rados.0.trial039.stdout:34: finishing write tid 3 to trial03942481-34 2026-02-25T08:41:50.719 INFO:tasks.rados.rados.0.trial039.stdout:34: finishing write tid 4 to trial03942481-34 2026-02-25T08:41:50.720 INFO:tasks.rados.rados.0.trial039.stdout:34: oid 34 updating version 0 to 7 2026-02-25T08:41:50.720 INFO:tasks.rados.rados.0.trial039.stdout:34: oid 34 updating version 7 to 9 2026-02-25T08:41:50.720 INFO:tasks.rados.rados.0.trial039.stdout:34: oid 34 version 9 is already newer than 9 2026-02-25T08:41:50.720 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2026-02-25T08:41:50.720 INFO:tasks.rados.rados.0.trial039.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2026-02-25T08:41:50.720 INFO:tasks.rados.rados.0.trial039.stdout:34: done (15 left) 2026-02-25T08:41:50.720 INFO:tasks.rados.rados.0.trial039.stdout:51: write (excl) oid 2 current snap is 0 2026-02-25T08:41:50.720 INFO:tasks.rados.rados.0.trial039.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2026-02-25T08:41:50.723 INFO:tasks.rados.rados.0.trial039.stdout:51: writing trial03942481-2 from 440350 to 900918 tid 1 2026-02-25T08:41:50.726 INFO:tasks.rados.rados.0.trial039.stdout:51: writing trial03942481-2 from 1309629 to 1852295 tid 2 2026-02-25T08:41:50.731 INFO:tasks.rados.rados.0.trial039.stdout:51: writing trial03942481-2 from 2323973 to 2840350 tid 3 2026-02-25T08:41:50.731 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.731 INFO:tasks.rados.rados.0.trial039.stdout:32: finishing write tid 2 to trial03942481-32 2026-02-25T08:41:50.731 INFO:tasks.rados.rados.0.trial039.stdout:48: finishing write tid 2 to trial03942481-48 2026-02-25T08:41:50.736 INFO:tasks.rados.rados.0.trial039.stdout:37: finishing write tid 2 to trial03942481-37 2026-02-25T08:41:50.738 INFO:tasks.rados.rados.0.trial039.stdout:32: finishing write tid 3 to trial03942481-32 2026-02-25T08:41:50.741 INFO:tasks.rados.rados.0.trial039.stdout:32: finishing write tid 4 to trial03942481-32 2026-02-25T08:41:50.741 INFO:tasks.rados.rados.0.trial039.stdout:32: oid 32 updating version 0 to 9 2026-02-25T08:41:50.741 INFO:tasks.rados.rados.0.trial039.stdout:32: oid 32 version 9 is already newer than 9 2026-02-25T08:41:50.742 INFO:tasks.rados.rados.0.trial039.stdout:32: oid 32 version 9 is already newer than 7 2026-02-25T08:41:50.742 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2026-02-25T08:41:50.742 INFO:tasks.rados.rados.0.trial039.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2026-02-25T08:41:50.742 INFO:tasks.rados.rados.0.trial039.stdout:32: done (15 left) 2026-02-25T08:41:50.742 INFO:tasks.rados.rados.0.trial039.stdout:52: write (excl) oid 14 current snap is 0 2026-02-25T08:41:50.742 INFO:tasks.rados.rados.0.trial039.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2026-02-25T08:41:50.745 INFO:tasks.rados.rados.0.trial039.stdout:52: writing trial03942481-14 from 457157 to 992974 tid 1 2026-02-25T08:41:50.750 INFO:tasks.rados.rados.0.trial039.stdout:52: writing trial03942481-14 from 1768111 to 2170788 tid 2 2026-02-25T08:41:50.750 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.750 INFO:tasks.rados.rados.0.trial039.stdout:48: finishing write tid 3 to trial03942481-48 2026-02-25T08:41:50.750 INFO:tasks.rados.rados.0.trial039.stdout:37: finishing write tid 3 to trial03942481-37 2026-02-25T08:41:50.751 INFO:tasks.rados.rados.0.trial039.stdout:48: finishing write tid 4 to trial03942481-48 2026-02-25T08:41:50.751 INFO:tasks.rados.rados.0.trial039.stdout:48: oid 48 updating version 0 to 6 2026-02-25T08:41:50.751 INFO:tasks.rados.rados.0.trial039.stdout:48: oid 48 version 6 is already newer than 6 2026-02-25T08:41:50.751 INFO:tasks.rados.rados.0.trial039.stdout:48: oid 48 version 6 is already newer than 4 2026-02-25T08:41:50.751 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 48 v 6 (ObjNum 47 snap 0 seq_num 47) dirty exists 2026-02-25T08:41:50.751 INFO:tasks.rados.rados.0.trial039.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:50.751 INFO:tasks.rados.rados.0.trial039.stdout:48: done (15 left) 2026-02-25T08:41:50.751 INFO:tasks.rados.rados.0.trial039.stdout:53: read oid 20 snap -1 2026-02-25T08:41:50.751 INFO:tasks.rados.rados.0.trial039.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:50.752 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.752 INFO:tasks.rados.rados.0.trial039.stdout:36: finishing write tid 1 to trial03942481-36 2026-02-25T08:41:50.753 INFO:tasks.rados.rados.0.trial039.stdout:37: finishing write tid 4 to trial03942481-37 2026-02-25T08:41:50.753 INFO:tasks.rados.rados.0.trial039.stdout:37: oid 37 updating version 0 to 10 2026-02-25T08:41:50.753 INFO:tasks.rados.rados.0.trial039.stdout:37: oid 37 updating version 10 to 12 2026-02-25T08:41:50.753 INFO:tasks.rados.rados.0.trial039.stdout:37: oid 37 version 12 is already newer than 12 2026-02-25T08:41:50.753 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 37 v 12 (ObjNum 36 snap 0 seq_num 36) dirty exists 2026-02-25T08:41:50.753 INFO:tasks.rados.rados.0.trial039.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2026-02-25T08:41:50.753 INFO:tasks.rados.rados.0.trial039.stdout:37: done (15 left) 2026-02-25T08:41:50.754 INFO:tasks.rados.rados.0.trial039.stdout:54: read oid 32 snap -1 2026-02-25T08:41:50.754 INFO:tasks.rados.rados.0.trial039.stdout:54: expect (ObjNum 31 snap 0 seq_num 31) 2026-02-25T08:41:50.754 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.762 INFO:tasks.rados.rados.0.trial039.stdout:43: finishing write tid 3 to trial03942481-43 2026-02-25T08:41:50.767 INFO:tasks.rados.rados.0.trial039.stdout:43: finishing write tid 4 to trial03942481-43 2026-02-25T08:41:50.767 INFO:tasks.rados.rados.0.trial039.stdout:43: oid 43 updating version 0 to 15 2026-02-25T08:41:50.767 INFO:tasks.rados.rados.0.trial039.stdout:43: oid 43 version 15 is already newer than 15 2026-02-25T08:41:50.768 INFO:tasks.rados.rados.0.trial039.stdout:43: oid 43 version 15 is already newer than 13 2026-02-25T08:41:50.768 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 43 v 15 (ObjNum 42 snap 0 seq_num 42) dirty exists 2026-02-25T08:41:50.768 INFO:tasks.rados.rados.0.trial039.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2026-02-25T08:41:50.768 INFO:tasks.rados.rados.0.trial039.stdout:43: done (15 left) 2026-02-25T08:41:50.768 INFO:tasks.rados.rados.0.trial039.stdout:55: read oid 24 snap -1 2026-02-25T08:41:50.768 INFO:tasks.rados.rados.0.trial039.stdout:55: expect (ObjNum 23 snap 0 seq_num 23) 2026-02-25T08:41:50.768 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.769 INFO:tasks.rados.rados.0.trial039.stdout:44: finishing write tid 1 to trial03942481-44 2026-02-25T08:41:50.771 INFO:tasks.rados.rados.0.trial039.stdout:40: finishing write tid 1 to trial03942481-40 2026-02-25T08:41:50.780 INFO:tasks.rados.rados.0.trial039.stdout:46: finishing write tid 2 to trial03942481-46 2026-02-25T08:41:50.793 INFO:tasks.rados.rados.0.trial039.stdout:31: finishing write tid 2 to trial03942481-31 2026-02-25T08:41:50.797 INFO:tasks.rados.rados.0.trial039.stdout:46: finishing write tid 3 to trial03942481-46 2026-02-25T08:41:50.798 INFO:tasks.rados.rados.0.trial039.stdout:40: finishing write tid 2 to trial03942481-40 2026-02-25T08:41:50.798 INFO:tasks.rados.rados.0.trial039.stdout:55: expect (ObjNum 23 snap 0 seq_num 23) 2026-02-25T08:41:50.813 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:50.833 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:50.856 INFO:tasks.rados.rados.0.trial039.stdout:55: done (15 left) 2026-02-25T08:41:50.856 INFO:tasks.rados.rados.0.trial039.stdout:56: read oid 27 snap -1 2026-02-25T08:41:50.856 INFO:tasks.rados.rados.0.trial039.stdout:56: expect (ObjNum 26 snap 0 seq_num 26) 2026-02-25T08:41:50.856 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.856 INFO:tasks.rados.rados.0.trial039.stdout:46: finishing write tid 4 to trial03942481-46 2026-02-25T08:41:50.856 INFO:tasks.rados.rados.0.trial039.stdout:46: oid 46 updating version 0 to 9 2026-02-25T08:41:50.856 INFO:tasks.rados.rados.0.trial039.stdout:46: oid 46 version 9 is already newer than 7 2026-02-25T08:41:50.856 INFO:tasks.rados.rados.0.trial039.stdout:46: oid 46 version 9 is already newer than 9 2026-02-25T08:41:50.856 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 46 v 9 (ObjNum 45 snap 0 seq_num 45) dirty exists 2026-02-25T08:41:50.856 INFO:tasks.rados.rados.0.trial039.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:47: finishing write tid 2 to trial03942481-47 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:47: finishing write tid 3 to trial03942481-47 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:46: done (15 left) 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:57: read oid 26 snap -1 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:57: expect (ObjNum 25 snap 0 seq_num 25) 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:40: finishing write tid 3 to trial03942481-40 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:47: finishing write tid 4 to trial03942481-47 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:47: oid 47 updating version 0 to 4 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:47: oid 47 updating version 4 to 6 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:47: oid 47 version 6 is already newer than 6 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 47 v 6 (ObjNum 46 snap 0 seq_num 46) dirty exists 2026-02-25T08:41:50.857 INFO:tasks.rados.rados.0.trial039.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:31: finishing write tid 3 to trial03942481-31 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:31: finishing write tid 4 to trial03942481-31 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:31: oid 31 updating version 0 to 15 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:31: oid 31 version 15 is already newer than 15 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:31: oid 31 version 15 is already newer than 13 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:40: finishing write tid 4 to trial03942481-40 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:40: oid 40 updating version 0 to 12 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:40: oid 40 version 12 is already newer than 12 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:40: oid 40 version 12 is already newer than 10 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 40 v 12 (ObjNum 39 snap 0 seq_num 39) dirty exists 2026-02-25T08:41:50.858 INFO:tasks.rados.rados.0.trial039.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:39: finishing write tid 1 to trial03942481-39 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:31: done (15 left) 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:40: done (14 left) 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:47: done (13 left) 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:58: read oid 10 snap -1 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:58: expect (ObjNum 9 snap 0 seq_num 9) 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:59: read oid 31 snap -1 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:59: expect (ObjNum 30 snap 0 seq_num 30) 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:60: read oid 15 snap -1 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:60: expect (ObjNum 14 snap 0 seq_num 14) 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.859 INFO:tasks.rados.rados.0.trial039.stdout:45: finishing write tid 1 to trial03942481-45 2026-02-25T08:41:50.860 INFO:tasks.rados.rados.0.trial039.stdout:36: finishing write tid 2 to trial03942481-36 2026-02-25T08:41:50.860 INFO:tasks.rados.rados.0.trial039.stdout:45: finishing write tid 2 to trial03942481-45 2026-02-25T08:41:50.860 INFO:tasks.rados.rados.0.trial039.stdout:45: finishing write tid 3 to trial03942481-45 2026-02-25T08:41:50.860 INFO:tasks.rados.rados.0.trial039.stdout:45: finishing write tid 4 to trial03942481-45 2026-02-25T08:41:50.861 INFO:tasks.rados.rados.0.trial039.stdout:45: oid 45 updating version 0 to 15 2026-02-25T08:41:50.861 INFO:tasks.rados.rados.0.trial039.stdout:45: oid 45 version 15 is already newer than 15 2026-02-25T08:41:50.861 INFO:tasks.rados.rados.0.trial039.stdout:45: oid 45 version 15 is already newer than 13 2026-02-25T08:41:50.861 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 45 v 15 (ObjNum 44 snap 0 seq_num 44) dirty exists 2026-02-25T08:41:50.861 INFO:tasks.rados.rados.0.trial039.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2026-02-25T08:41:50.861 INFO:tasks.rados.rados.0.trial039.stdout:45: done (15 left) 2026-02-25T08:41:50.861 INFO:tasks.rados.rados.0.trial039.stdout:61: write (excl) oid 24 current snap is 0 2026-02-25T08:41:50.861 INFO:tasks.rados.rados.0.trial039.stdout:61: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2026-02-25T08:41:50.862 INFO:tasks.rados.rados.0.trial039.stdout:61: writing trial03942481-24 from 473964 to 1085030 tid 1 2026-02-25T08:41:50.867 INFO:tasks.rados.rados.0.trial039.stdout:61: writing trial03942481-24 from 1826593 to 2372928 tid 2 2026-02-25T08:41:50.869 INFO:tasks.rados.rados.0.trial039.stdout:61: writing trial03942481-24 from 2778819 to 2873964 tid 3 2026-02-25T08:41:50.869 INFO:tasks.rados.rados.0.trial039.stdout: waiting on 16 2026-02-25T08:41:50.869 INFO:tasks.rados.rados.0.trial039.stdout:36: finishing write tid 3 to trial03942481-36 2026-02-25T08:41:50.869 INFO:tasks.rados.rados.0.trial039.stdout:50: finishing write tid 1 to trial03942481-50 2026-02-25T08:41:50.873 INFO:tasks.rados.rados.0.trial039.stdout:36: finishing write tid 4 to trial03942481-36 2026-02-25T08:41:50.873 INFO:tasks.rados.rados.0.trial039.stdout:36: oid 36 updating version 0 to 12 2026-02-25T08:41:50.873 INFO:tasks.rados.rados.0.trial039.stdout:36: oid 36 version 12 is already newer than 12 2026-02-25T08:41:50.873 INFO:tasks.rados.rados.0.trial039.stdout:36: oid 36 version 12 is already newer than 10 2026-02-25T08:41:50.873 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 36 v 12 (ObjNum 35 snap 0 seq_num 35) dirty exists 2026-02-25T08:41:50.873 INFO:tasks.rados.rados.0.trial039.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:41:50.873 INFO:tasks.rados.rados.0.trial039.stdout:36: done (15 left) 2026-02-25T08:41:50.874 INFO:tasks.rados.rados.0.trial039.stdout:62: snap_create 2026-02-25T08:41:50.879 INFO:tasks.rados.rados.0.trial039.stdout:44: finishing write tid 2 to trial03942481-44 2026-02-25T08:41:50.887 INFO:tasks.rados.rados.0.trial039.stdout:58: expect (ObjNum 9 snap 0 seq_num 9) 2026-02-25T08:41:50.888 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:50.911 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1178484 2026-02-25T08:41:50.932 INFO:tasks.rados.rados.0.trial039.stdout:60: expect (ObjNum 14 snap 0 seq_num 14) 2026-02-25T08:41:50.965 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:50.985 INFO:tasks.rados.rados.0.trial039.stdout:50: finishing write tid 2 to trial03942481-50 2026-02-25T08:41:50.985 INFO:tasks.rados.rados.0.trial039.stdout:44: finishing write tid 3 to trial03942481-44 2026-02-25T08:41:50.985 INFO:tasks.rados.rados.0.trial039.stdout:44: finishing write tid 4 to trial03942481-44 2026-02-25T08:41:50.985 INFO:tasks.rados.rados.0.trial039.stdout:44: oid 44 updating version 0 to 15 2026-02-25T08:41:50.985 INFO:tasks.rados.rados.0.trial039.stdout:44: oid 44 version 15 is already newer than 13 2026-02-25T08:41:50.985 INFO:tasks.rados.rados.0.trial039.stdout:44: oid 44 version 15 is already newer than 15 2026-02-25T08:41:50.985 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 15 (ObjNum 43 snap 0 seq_num 43) dirty exists 2026-02-25T08:41:50.985 INFO:tasks.rados.rados.0.trial039.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:41:50.985 INFO:tasks.rados.rados.0.trial039.stdout:50: finishing write tid 3 to trial03942481-50 2026-02-25T08:41:50.986 INFO:tasks.rados.rados.0.trial039.stdout:51: finishing write tid 1 to trial03942481-2 2026-02-25T08:41:50.986 INFO:tasks.rados.rados.0.trial039.stdout:57: expect (ObjNum 25 snap 0 seq_num 25) 2026-02-25T08:41:51.041 INFO:tasks.rados.rados.0.trial039.stdout:50: finishing write tid 4 to trial03942481-50 2026-02-25T08:41:51.041 INFO:tasks.rados.rados.0.trial039.stdout:50: oid 50 updating version 0 to 18 2026-02-25T08:41:51.041 INFO:tasks.rados.rados.0.trial039.stdout:50: oid 50 version 18 is already newer than 18 2026-02-25T08:41:51.041 INFO:tasks.rados.rados.0.trial039.stdout:50: oid 50 version 18 is already newer than 16 2026-02-25T08:41:51.041 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 50 v 18 (ObjNum 49 snap 0 seq_num 49) dirty exists 2026-02-25T08:41:51.041 INFO:tasks.rados.rados.0.trial039.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2026-02-25T08:41:51.042 INFO:tasks.rados.rados.0.trial039.stdout:52: finishing write tid 1 to trial03942481-14 2026-02-25T08:41:51.042 INFO:tasks.rados.rados.0.trial039.stdout:39: finishing write tid 2 to trial03942481-39 2026-02-25T08:41:51.042 INFO:tasks.rados.rados.0.trial039.stdout:61: finishing write tid 1 to trial03942481-24 2026-02-25T08:41:51.042 INFO:tasks.rados.rados.0.trial039.stdout:39: finishing write tid 3 to trial03942481-39 2026-02-25T08:41:51.042 INFO:tasks.rados.rados.0.trial039.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:51.042 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:51.061 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:51.081 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:51.101 INFO:tasks.rados.rados.0.trial039.stdout:39: finishing write tid 4 to trial03942481-39 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:39: oid 39 updating version 0 to 18 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:39: oid 39 version 18 is already newer than 18 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:39: oid 39 version 18 is already newer than 16 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 39 v 18 (ObjNum 38 snap 0 seq_num 38) dirty exists 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:52: finishing write tid 2 to trial03942481-14 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:42: finishing write tid 1 to trial03942481-42 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:52: finishing write tid 3 to trial03942481-14 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:52: finishing write tid 4 to trial03942481-14 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:52: oid 14 updating version 0 to 21 2026-02-25T08:41:51.102 INFO:tasks.rados.rados.0.trial039.stdout:52: oid 14 version 21 is already newer than 21 2026-02-25T08:41:51.103 INFO:tasks.rados.rados.0.trial039.stdout:52: oid 14 version 21 is already newer than 19 2026-02-25T08:41:51.103 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 14 v 21 (ObjNum 51 snap 0 seq_num 51) dirty exists 2026-02-25T08:41:51.103 INFO:tasks.rados.rados.0.trial039.stdout:52: left oid 14 (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:41:51.103 INFO:tasks.rados.rados.0.trial039.stdout:61: finishing write tid 2 to trial03942481-24 2026-02-25T08:41:51.103 INFO:tasks.rados.rados.0.trial039.stdout:61: finishing write tid 3 to trial03942481-24 2026-02-25T08:41:51.103 INFO:tasks.rados.rados.0.trial039.stdout:61: finishing write tid 4 to trial03942481-24 2026-02-25T08:41:51.103 INFO:tasks.rados.rados.0.trial039.stdout:59: expect (ObjNum 30 snap 0 seq_num 30) 2026-02-25T08:41:51.121 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1538120 2026-02-25T08:41:51.149 INFO:tasks.rados.rados.0.trial039.stdout:61: finishing write tid 5 to trial03942481-24 2026-02-25T08:41:51.149 INFO:tasks.rados.rados.0.trial039.stdout:61: oid 24 updating version 0 to 10 2026-02-25T08:41:51.149 INFO:tasks.rados.rados.0.trial039.stdout:61: oid 24 version 10 is already newer than 10 2026-02-25T08:41:51.150 INFO:tasks.rados.rados.0.trial039.stdout:61: oid 24 version 10 is already newer than 10 2026-02-25T08:41:51.150 INFO:tasks.rados.rados.0.trial039.stdout:61: oid 24 version 10 is already newer than 7 2026-02-25T08:41:51.150 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 24 v 10 (ObjNum 52 snap 0 seq_num 52) dirty exists 2026-02-25T08:41:51.150 INFO:tasks.rados.rados.0.trial039.stdout:61: left oid 24 (ObjNum 52 snap 0 seq_num 52) 2026-02-25T08:41:51.150 INFO:tasks.rados.rados.0.trial039.stdout:49: finishing write tid 1 to trial03942481-49 2026-02-25T08:41:51.150 INFO:tasks.rados.rados.0.trial039.stdout:42: finishing write tid 2 to trial03942481-42 2026-02-25T08:41:51.150 INFO:tasks.rados.rados.0.trial039.stdout:42: finishing write tid 3 to trial03942481-42 2026-02-25T08:41:51.150 INFO:tasks.rados.rados.0.trial039.stdout:54: expect (ObjNum 31 snap 0 seq_num 31) 2026-02-25T08:41:51.164 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:51.183 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:51.201 INFO:tasks.rados.rados.0.trial039.stdout:42: finishing write tid 4 to trial03942481-42 2026-02-25T08:41:51.201 INFO:tasks.rados.rados.0.trial039.stdout:42: oid 42 updating version 0 to 21 2026-02-25T08:41:51.201 INFO:tasks.rados.rados.0.trial039.stdout:42: oid 42 version 21 is already newer than 19 2026-02-25T08:41:51.201 INFO:tasks.rados.rados.0.trial039.stdout:42: oid 42 version 21 is already newer than 21 2026-02-25T08:41:51.201 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 42 v 21 (ObjNum 41 snap 0 seq_num 41) dirty exists 2026-02-25T08:41:51.201 INFO:tasks.rados.rados.0.trial039.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2026-02-25T08:41:51.201 INFO:tasks.rados.rados.0.trial039.stdout:56: expect (ObjNum 26 snap 0 seq_num 26) 2026-02-25T08:41:51.205 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1103577 2026-02-25T08:41:51.216 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:51.259 INFO:tasks.rados.rados.0.trial039.stdout:49: finishing write tid 2 to trial03942481-49 2026-02-25T08:41:51.259 INFO:tasks.rados.rados.0.trial039.stdout:49: finishing write tid 3 to trial03942481-49 2026-02-25T08:41:51.259 INFO:tasks.rados.rados.0.trial039.stdout:49: finishing write tid 4 to trial03942481-49 2026-02-25T08:41:51.259 INFO:tasks.rados.rados.0.trial039.stdout:49: oid 49 updating version 0 to 13 2026-02-25T08:41:51.259 INFO:tasks.rados.rados.0.trial039.stdout:49: oid 49 updating version 13 to 15 2026-02-25T08:41:51.259 INFO:tasks.rados.rados.0.trial039.stdout:49: oid 49 version 15 is already newer than 15 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:51: finishing write tid 2 to trial03942481-2 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:51: finishing write tid 3 to trial03942481-2 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:51: finishing write tid 4 to trial03942481-2 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:51: finishing write tid 5 to trial03942481-2 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:51: oid 2 updating version 0 to 16 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:51: oid 2 updating version 16 to 19 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:51: oid 2 version 19 is already newer than 19 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:51: oid 2 version 19 is already newer than 19 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:41:51.260 INFO:tasks.rados.rados.0.trial039.stdout:39: done (15 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:42: done (14 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:44: done (13 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:49: done (12 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:50: done (11 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:51: done (10 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:52: done (9 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:53: done (8 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:54: done (7 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:56: done (6 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:57: done (5 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:58: done (4 left) 2026-02-25T08:41:51.261 INFO:tasks.rados.rados.0.trial039.stdout:59: done (3 left) 2026-02-25T08:41:51.262 INFO:tasks.rados.rados.0.trial039.stdout:60: done (2 left) 2026-02-25T08:41:51.262 INFO:tasks.rados.rados.0.trial039.stdout:61: done (1 left) 2026-02-25T08:41:51.262 INFO:tasks.rados.rados.0.trial039.stdout:62: done (0 left) 2026-02-25T08:41:51.262 INFO:tasks.rados.rados.0.trial039.stdout:63: write oid 45 current snap is 1 2026-02-25T08:41:51.262 INFO:tasks.rados.rados.0.trial039.stdout:63: seq_num 53 ranges {490771=686315,1885075=406346} 2026-02-25T08:41:51.262 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:51.257+0000 7f096afcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:51.263 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:51.260+0000 7f096afcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:51.263 INFO:tasks.rados.rados.0.trial039.stdout:63: writing trial03942481-45 from 490771 to 1177086 tid 1 2026-02-25T08:41:51.266 INFO:tasks.rados.rados.0.trial039.stdout:63: writing trial03942481-45 from 1885075 to 2291421 tid 2 2026-02-25T08:41:51.267 INFO:tasks.rados.rados.0.trial039.stdout:64: read oid 16 snap -1 2026-02-25T08:41:51.267 INFO:tasks.rados.rados.0.trial039.stdout:64: expect (ObjNum 15 snap 0 seq_num 15) 2026-02-25T08:41:51.267 INFO:tasks.rados.rados.0.trial039.stdout:65: write oid 44 current snap is 1 2026-02-25T08:41:51.267 INFO:tasks.rados.rados.0.trial039.stdout:65: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2026-02-25T08:41:51.270 INFO:tasks.rados.rados.0.trial039.stdout:65: writing trial03942481-44 from 507578 to 985495 tid 1 2026-02-25T08:41:51.274 INFO:tasks.rados.rados.0.trial039.stdout:65: writing trial03942481-44 from 1543557 to 2093561 tid 2 2026-02-25T08:41:51.277 INFO:tasks.rados.rados.0.trial039.stdout:65: writing trial03942481-44 from 2833665 to 2907578 tid 3 2026-02-25T08:41:51.278 INFO:tasks.rados.rados.0.trial039.stdout:66: read oid 48 snap -1 2026-02-25T08:41:51.278 INFO:tasks.rados.rados.0.trial039.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:51.278 INFO:tasks.rados.rados.0.trial039.stdout:67: delete oid 31 current snap is 1 2026-02-25T08:41:51.291 INFO:tasks.rados.rados.0.trial039.stdout:63: finishing write tid 1 to trial03942481-45 2026-02-25T08:41:51.292 INFO:tasks.rados.rados.0.trial039.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:51.292 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:51.305 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:41:51.321 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:41:51.322 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:41:51.322 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1154049 2026-02-25T08:41:51.338 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 999652 2026-02-25T08:41:51.353 INFO:tasks.rados.rados.0.trial039.stdout:64: expect (ObjNum 15 snap 0 seq_num 15) 2026-02-25T08:41:51.353 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:51.411 INFO:tasks.rados.rados.0.trial039.stdout:63: finishing write tid 2 to trial03942481-45 2026-02-25T08:41:51.412 INFO:tasks.rados.rados.0.trial039.stdout:63: finishing write tid 3 to trial03942481-45 2026-02-25T08:41:51.412 INFO:tasks.rados.rados.0.trial039.stdout:63: finishing write tid 4 to trial03942481-45 2026-02-25T08:41:51.412 INFO:tasks.rados.rados.0.trial039.stdout:63: oid 45 updating version 0 to 25 2026-02-25T08:41:51.412 INFO:tasks.rados.rados.0.trial039.stdout:63: oid 45 version 25 is already newer than 25 2026-02-25T08:41:51.412 INFO:tasks.rados.rados.0.trial039.stdout:63: oid 45 version 25 is already newer than 25 2026-02-25T08:41:51.412 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 45 v 25 (ObjNum 53 snap 1 seq_num 53) dirty exists 2026-02-25T08:41:51.412 INFO:tasks.rados.rados.0.trial039.stdout:63: left oid 45 (ObjNum 53 snap 1 seq_num 53) 2026-02-25T08:41:51.412 INFO:tasks.rados.rados.0.trial039.stdout:65: finishing write tid 1 to trial03942481-44 2026-02-25T08:41:51.412 INFO:tasks.rados.rados.0.trial039.stdout:65: finishing write tid 2 to trial03942481-44 2026-02-25T08:41:51.412 INFO:tasks.rados.rados.0.trial039.stdout:63: done (4 left) 2026-02-25T08:41:51.413 INFO:tasks.rados.rados.0.trial039.stdout:64: done (3 left) 2026-02-25T08:41:51.413 INFO:tasks.rados.rados.0.trial039.stdout:66: done (2 left) 2026-02-25T08:41:51.413 INFO:tasks.rados.rados.0.trial039.stdout:67: done (1 left) 2026-02-25T08:41:51.413 INFO:tasks.rados.rados.0.trial039.stdout:68: snap_remove snap 0 2026-02-25T08:41:51.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~17s 2026-02-25T08:41:52.156 INFO:tasks.rados.rados.0.trial039.stdout:68: done (1 left) 2026-02-25T08:41:52.156 INFO:tasks.rados.rados.0.trial039.stdout:69: read oid 6 snap -1 2026-02-25T08:41:52.156 INFO:tasks.rados.rados.0.trial039.stdout:69: expect (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:41:52.156 INFO:tasks.rados.rados.0.trial039.stdout:70: read oid 36 snap -1 2026-02-25T08:41:52.156 INFO:tasks.rados.rados.0.trial039.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:41:52.156 INFO:tasks.rados.rados.0.trial039.stdout:71: read oid 48 snap -1 2026-02-25T08:41:52.156 INFO:tasks.rados.rados.0.trial039.stdout:71: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:52.156 INFO:tasks.rados.rados.0.trial039.stdout:72: read oid 15 snap -1 2026-02-25T08:41:52.156 INFO:tasks.rados.rados.0.trial039.stdout:72: expect (ObjNum 14 snap 0 seq_num 14) 2026-02-25T08:41:52.156 INFO:tasks.rados.rados.0.trial039.stdout:73: read oid 19 snap -1 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:73: expect (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:74: write oid 29 current snap is 1 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:65: finishing write tid 3 to trial03942481-44 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:65: finishing write tid 4 to trial03942481-44 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:65: finishing write tid 5 to trial03942481-44 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:65: oid 44 updating version 0 to 24 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:65: oid 44 version 24 is already newer than 24 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:65: oid 44 version 24 is already newer than 24 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:65: oid 44 version 24 is already newer than 24 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 24 (ObjNum 54 snap 1 seq_num 54) dirty exists 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:65: left oid 44 (ObjNum 54 snap 1 seq_num 54) 2026-02-25T08:41:52.157 INFO:tasks.rados.rados.0.trial039.stdout:74: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2026-02-25T08:41:52.159 INFO:tasks.rados.rados.0.trial039.stdout:74: writing trial03942481-29 from 541192 to 1169607 tid 1 2026-02-25T08:41:52.163 INFO:tasks.rados.rados.0.trial039.stdout:74: writing trial03942481-29 from 1660521 to 2214194 tid 2 2026-02-25T08:41:52.167 INFO:tasks.rados.rados.0.trial039.stdout:74: writing trial03942481-29 from 2888511 to 2941192 tid 3 2026-02-25T08:41:52.167 INFO:tasks.rados.rados.0.trial039.stdout:65: done (6 left) 2026-02-25T08:41:52.167 INFO:tasks.rados.rados.0.trial039.stdout:75: read oid 37 snap -1 2026-02-25T08:41:52.167 INFO:tasks.rados.rados.0.trial039.stdout:75: expect (ObjNum 36 snap 0 seq_num 36) 2026-02-25T08:41:52.167 INFO:tasks.rados.rados.0.trial039.stdout:76: read oid 33 snap -1 2026-02-25T08:41:52.167 INFO:tasks.rados.rados.0.trial039.stdout:76: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-25T08:41:52.167 INFO:tasks.rados.rados.0.trial039.stdout:77: write (excl) oid 24 current snap is 1 2026-02-25T08:41:52.167 INFO:tasks.rados.rados.0.trial039.stdout:77: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2026-02-25T08:41:52.172 INFO:tasks.rados.rados.0.trial039.stdout:77: writing trial03942481-24 from 557999 to 1261663 tid 1 2026-02-25T08:41:52.176 INFO:tasks.rados.rados.0.trial039.stdout:77: writing trial03942481-24 from 1719003 to 2132687 tid 2 2026-02-25T08:41:52.179 INFO:tasks.rados.rados.0.trial039.stdout:77: writing trial03942481-24 from 2632287 to 2957999 tid 3 2026-02-25T08:41:52.179 INFO:tasks.rados.rados.0.trial039.stdout:78: write oid 49 current snap is 1 2026-02-25T08:41:52.180 INFO:tasks.rados.rados.0.trial039.stdout:78: seq_num 58 ranges {574806=778913,2061132=557342} 2026-02-25T08:41:52.185 INFO:tasks.rados.rados.0.trial039.stdout:78: writing trial03942481-49 from 574806 to 1353719 tid 1 2026-02-25T08:41:52.191 INFO:tasks.rados.rados.0.trial039.stdout:78: writing trial03942481-49 from 2061132 to 2618474 tid 2 2026-02-25T08:41:52.191 INFO:tasks.rados.rados.0.trial039.stdout:79: delete oid 43 current snap is 1 2026-02-25T08:41:52.191 INFO:tasks.rados.rados.0.trial039.stdout:72: expect (ObjNum 14 snap 0 seq_num 14) 2026-02-25T08:41:52.214 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 639937 2026-02-25T08:41:52.228 INFO:tasks.rados.rados.0.trial039.stdout:71: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:52.228 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:52.298 INFO:tasks.rados.rados.0.trial039.stdout:76: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-25T08:41:52.332 INFO:tasks.rados.rados.0.trial039.stdout:69: expect (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:41:52.351 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1308088 2026-02-25T08:41:52.363 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1664868 2026-02-25T08:41:52.371 INFO:tasks.rados.rados.0.trial039.stdout:75: expect (ObjNum 36 snap 0 seq_num 36) 2026-02-25T08:41:52.413 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2163643 2026-02-25T08:41:52.416 INFO:tasks.rados.rados.0.trial039.stdout:73: expect (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:41:52.416 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 651661 2026-02-25T08:41:52.432 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:52.452 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:52.472 INFO:tasks.rados.rados.0.trial039.stdout:74: finishing write tid 1 to trial03942481-29 2026-02-25T08:41:52.472 INFO:tasks.rados.rados.0.trial039.stdout:69: done (10 left) 2026-02-25T08:41:52.472 INFO:tasks.rados.rados.0.trial039.stdout:71: done (9 left) 2026-02-25T08:41:52.473 INFO:tasks.rados.rados.0.trial039.stdout:72: done (8 left) 2026-02-25T08:41:52.473 INFO:tasks.rados.rados.0.trial039.stdout:73: done (7 left) 2026-02-25T08:41:52.473 INFO:tasks.rados.rados.0.trial039.stdout:75: done (6 left) 2026-02-25T08:41:52.473 INFO:tasks.rados.rados.0.trial039.stdout:76: done (5 left) 2026-02-25T08:41:52.473 INFO:tasks.rados.rados.0.trial039.stdout:79: done (4 left) 2026-02-25T08:41:52.473 INFO:tasks.rados.rados.0.trial039.stdout:80: snap_create 2026-02-25T08:41:52.473 INFO:tasks.rados.rados.0.trial039.stdout:70: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:41:52.484 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1456490 2026-02-25T08:41:52.498 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:52.521 INFO:tasks.rados.rados.0.trial039.stdout:74: finishing write tid 2 to trial03942481-29 2026-02-25T08:41:52.521 INFO:tasks.rados.rados.0.trial039.stdout:74: finishing write tid 3 to trial03942481-29 2026-02-25T08:41:52.521 INFO:tasks.rados.rados.0.trial039.stdout:74: finishing write tid 4 to trial03942481-29 2026-02-25T08:41:52.521 INFO:tasks.rados.rados.0.trial039.stdout:74: finishing write tid 5 to trial03942481-29 2026-02-25T08:41:52.521 INFO:tasks.rados.rados.0.trial039.stdout:74: oid 29 updating version 0 to 13 2026-02-25T08:41:52.521 INFO:tasks.rados.rados.0.trial039.stdout:74: oid 29 version 13 is already newer than 13 2026-02-25T08:41:52.521 INFO:tasks.rados.rados.0.trial039.stdout:74: oid 29 version 13 is already newer than 13 2026-02-25T08:41:52.521 INFO:tasks.rados.rados.0.trial039.stdout:74: oid 29 version 13 is already newer than 13 2026-02-25T08:41:52.521 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 29 v 13 (ObjNum 56 snap 1 seq_num 56) dirty exists 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:74: left oid 29 (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:77: finishing write tid 1 to trial03942481-24 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:78: finishing write tid 1 to trial03942481-49 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:78: finishing write tid 2 to trial03942481-49 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:78: finishing write tid 3 to trial03942481-49 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:78: finishing write tid 4 to trial03942481-49 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:78: oid 49 updating version 0 to 18 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:78: oid 49 version 18 is already newer than 18 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:78: oid 49 version 18 is already newer than 18 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 49 v 18 (ObjNum 58 snap 1 seq_num 58) dirty exists 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:78: left oid 49 (ObjNum 58 snap 1 seq_num 58) 2026-02-25T08:41:52.522 INFO:tasks.rados.rados.0.trial039.stdout:77: finishing write tid 2 to trial03942481-24 2026-02-25T08:41:52.523 INFO:tasks.rados.rados.0.trial039.stdout:77: finishing write tid 3 to trial03942481-24 2026-02-25T08:41:52.523 INFO:tasks.rados.rados.0.trial039.stdout:77: finishing write tid 4 to trial03942481-24 2026-02-25T08:41:52.523 INFO:tasks.rados.rados.0.trial039.stdout:77: finishing write tid 5 to trial03942481-24 2026-02-25T08:41:52.523 INFO:tasks.rados.rados.0.trial039.stdout:77: oid 24 updating version 0 to 14 2026-02-25T08:41:52.523 INFO:tasks.rados.rados.0.trial039.stdout:77: oid 24 version 14 is already newer than 14 2026-02-25T08:41:52.523 INFO:tasks.rados.rados.0.trial039.stdout:77: oid 24 version 14 is already newer than 11 2026-02-25T08:41:52.523 INFO:tasks.rados.rados.0.trial039.stdout:77: oid 24 version 14 is already newer than 14 2026-02-25T08:41:52.523 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 24 v 14 (ObjNum 57 snap 1 seq_num 57) dirty exists 2026-02-25T08:41:52.523 INFO:tasks.rados.rados.0.trial039.stdout:77: left oid 24 (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:41:53.158 INFO:tasks.rados.rados.0.trial039.stdout:70: done (4 left) 2026-02-25T08:41:53.158 INFO:tasks.rados.rados.0.trial039.stdout:74: done (3 left) 2026-02-25T08:41:53.159 INFO:tasks.rados.rados.0.trial039.stdout:77: done (2 left) 2026-02-25T08:41:53.159 INFO:tasks.rados.rados.0.trial039.stdout:78: done (1 left) 2026-02-25T08:41:53.159 INFO:tasks.rados.rados.0.trial039.stdout:80: done (0 left) 2026-02-25T08:41:53.159 INFO:tasks.rados.rados.0.trial039.stdout:81: delete oid 42 current snap is 2 2026-02-25T08:41:53.191 INFO:tasks.rados.rados.0.trial039.stdout:81: done (0 left) 2026-02-25T08:41:53.192 INFO:tasks.rados.rados.0.trial039.stdout:82: read oid 48 snap 1 2026-02-25T08:41:53.192 INFO:tasks.rados.rados.0.trial039.stdout:82: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:53.192 INFO:tasks.rados.rados.0.trial039.stdout:83: delete oid 5 current snap is 2 2026-02-25T08:41:53.210 INFO:tasks.rados.rados.0.trial039.stdout:82: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:53.238 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:53.262 INFO:tasks.rados.rados.0.trial039.stdout:82: done (1 left) 2026-02-25T08:41:53.262 INFO:tasks.rados.rados.0.trial039.stdout:83: done (0 left) 2026-02-25T08:41:53.262 INFO:tasks.rados.rados.0.trial039.stdout:84: snap_create 2026-02-25T08:41:53.323 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:53.400 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:53.397+0000 7f5bb99cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:53.402 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:53.401+0000 7f5bb99cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:53.432 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:41:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:41:53.449 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:41:54.059 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:41:54.132 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:54.135 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:54.136 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:41:54.158 INFO:tasks.rados.rados.0.trial039.stdout:84: done (0 left) 2026-02-25T08:41:54.159 INFO:tasks.rados.rados.0.trial039.stdout:85: delete oid 19 current snap is 3 2026-02-25T08:41:54.192 INFO:tasks.rados.rados.0.trial039.stdout:85: done (0 left) 2026-02-25T08:41:54.192 INFO:tasks.rados.rados.0.trial039.stdout:86: read oid 2 snap -1 2026-02-25T08:41:54.192 INFO:tasks.rados.rados.0.trial039.stdout:86: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:41:54.192 INFO:tasks.rados.rados.0.trial039.stdout:87: write (excl) oid 50 current snap is 3 2026-02-25T08:41:54.193 INFO:tasks.rados.rados.0.trial039.stdout:87: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2026-02-25T08:41:54.196 INFO:tasks.rados.rados.0.trial039.stdout:87: writing trial03942481-50 from 658841 to 1130352 tid 1 2026-02-25T08:41:54.201 INFO:tasks.rados.rados.0.trial039.stdout:87: writing trial03942481-50 from 1553542 to 2261880 tid 2 2026-02-25T08:41:54.205 INFO:tasks.rados.rados.0.trial039.stdout:87: writing trial03942481-50 from 2964119 to 3058841 tid 3 2026-02-25T08:41:54.205 INFO:tasks.rados.rados.0.trial039.stdout:88: read oid 6 snap -1 2026-02-25T08:41:54.205 INFO:tasks.rados.rados.0.trial039.stdout:88: expect (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:41:54.205 INFO:tasks.rados.rados.0.trial039.stdout:89: snap_remove snap 2 2026-02-25T08:41:55.164 INFO:tasks.rados.rados.0.trial039.stdout:89: done (3 left) 2026-02-25T08:41:55.164 INFO:tasks.rados.rados.0.trial039.stdout:90: read oid 8 snap -1 2026-02-25T08:41:55.164 INFO:tasks.rados.rados.0.trial039.stdout:90: expect (ObjNum 7 snap 0 seq_num 7) 2026-02-25T08:41:55.164 INFO:tasks.rados.rados.0.trial039.stdout:91: read oid 14 snap -1 2026-02-25T08:41:55.164 INFO:tasks.rados.rados.0.trial039.stdout:91: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:41:55.164 INFO:tasks.rados.rados.0.trial039.stdout:92: snap_create 2026-02-25T08:41:55.164 INFO:tasks.rados.rados.0.trial039.stdout:86: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:41:55.164 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:55.240 INFO:tasks.rados.rados.0.trial039.stdout:88: expect (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:41:55.259 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 516662 2026-02-25T08:41:55.291 INFO:tasks.rados.rados.0.trial039.stdout:87: finishing write tid 1 to trial03942481-50 2026-02-25T08:41:55.291 INFO:tasks.rados.rados.0.trial039.stdout:87: finishing write tid 2 to trial03942481-50 2026-02-25T08:41:55.291 INFO:tasks.rados.rados.0.trial039.stdout:87: finishing write tid 3 to trial03942481-50 2026-02-25T08:41:55.291 INFO:tasks.rados.rados.0.trial039.stdout:87: finishing write tid 4 to trial03942481-50 2026-02-25T08:41:55.291 INFO:tasks.rados.rados.0.trial039.stdout:87: finishing write tid 5 to trial03942481-50 2026-02-25T08:41:55.291 INFO:tasks.rados.rados.0.trial039.stdout:87: oid 50 updating version 0 to 30 2026-02-25T08:41:55.292 INFO:tasks.rados.rados.0.trial039.stdout:87: oid 50 version 30 is already newer than 30 2026-02-25T08:41:55.292 INFO:tasks.rados.rados.0.trial039.stdout:87: oid 50 version 30 is already newer than 27 2026-02-25T08:41:55.292 INFO:tasks.rados.rados.0.trial039.stdout:87: oid 50 version 30 is already newer than 30 2026-02-25T08:41:55.292 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 50 v 30 (ObjNum 63 snap 3 seq_num 63) dirty exists 2026-02-25T08:41:55.292 INFO:tasks.rados.rados.0.trial039.stdout:87: left oid 50 (ObjNum 63 snap 3 seq_num 63) 2026-02-25T08:41:55.292 INFO:tasks.rados.rados.0.trial039.stdout:91: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:41:55.360 INFO:tasks.rados.rados.0.trial039.stdout:90: expect (ObjNum 7 snap 0 seq_num 7) 2026-02-25T08:41:55.378 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:55.396 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:41:55.450 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:55.525 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:55.523+0000 7f4250c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:55.528 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:55.526+0000 7f4250c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:55.552 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:41:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:41:55.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:41:56.166 INFO:tasks.rados.rados.0.trial039.stdout:86: done (5 left) 2026-02-25T08:41:56.166 INFO:tasks.rados.rados.0.trial039.stdout:87: done (4 left) 2026-02-25T08:41:56.166 INFO:tasks.rados.rados.0.trial039.stdout:88: done (3 left) 2026-02-25T08:41:56.167 INFO:tasks.rados.rados.0.trial039.stdout:90: done (2 left) 2026-02-25T08:41:56.167 INFO:tasks.rados.rados.0.trial039.stdout:91: done (1 left) 2026-02-25T08:41:56.167 INFO:tasks.rados.rados.0.trial039.stdout:92: done (0 left) 2026-02-25T08:41:56.167 INFO:tasks.rados.rados.0.trial039.stdout:93: write oid 42 current snap is 4 2026-02-25T08:41:56.167 INFO:tasks.rados.rados.0.trial039.stdout:93: seq_num 64 ranges {675648=546760,2012024=568349} 2026-02-25T08:41:56.170 INFO:tasks.rados.rados.0.trial039.stdout:93: writing trial03942481-42 from 675648 to 1222408 tid 1 2026-02-25T08:41:56.176 INFO:tasks.rados.rados.0.trial039.stdout:93: writing trial03942481-42 from 2012024 to 2580373 tid 2 2026-02-25T08:41:56.176 INFO:tasks.rados.rados.0.trial039.stdout:94: snap_remove snap 3 2026-02-25T08:41:57.171 INFO:tasks.rados.rados.0.trial039.stdout:94: done (1 left) 2026-02-25T08:41:57.171 INFO:tasks.rados.rados.0.trial039.stdout:95: snap_remove snap 1 2026-02-25T08:41:57.351 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~22s 2026-02-25T08:41:57.568 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:57.646 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:57.643+0000 7fef5bf8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:57.648 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:57.647+0000 7fef5bf8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:57.674 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:41:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:41:57.691 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:41:58.173 INFO:tasks.rados.rados.0.trial039.stdout:95: done (1 left) 2026-02-25T08:41:58.173 INFO:tasks.rados.rados.0.trial039.stdout:96: write (excl) oid 41 current snap is 4 2026-02-25T08:41:58.173 INFO:tasks.rados.rados.0.trial039.stdout:93: finishing write tid 1 to trial03942481-42 2026-02-25T08:41:58.173 INFO:tasks.rados.rados.0.trial039.stdout:96: seq_num 65 ranges {692455=622009,2070506=712007} 2026-02-25T08:41:58.177 INFO:tasks.rados.rados.0.trial039.stdout:96: writing trial03942481-41 from 692455 to 1314464 tid 1 2026-02-25T08:41:58.184 INFO:tasks.rados.rados.0.trial039.stdout:96: writing trial03942481-41 from 2070506 to 2782513 tid 2 2026-02-25T08:41:58.184 INFO:tasks.rados.rados.0.trial039.stdout:97: write oid 35 current snap is 4 2026-02-25T08:41:58.184 INFO:tasks.rados.rados.0.trial039.stdout:97: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2026-02-25T08:41:58.189 INFO:tasks.rados.rados.0.trial039.stdout:97: writing trial03942481-35 from 709262 to 1406520 tid 1 2026-02-25T08:41:58.194 INFO:tasks.rados.rados.0.trial039.stdout:97: writing trial03942481-35 from 2012635 to 2584653 tid 2 2026-02-25T08:41:58.196 INFO:tasks.rados.rados.0.trial039.stdout:97: writing trial03942481-35 from 3046388 to 3109262 tid 3 2026-02-25T08:41:58.196 INFO:tasks.rados.rados.0.trial039.stdout:98: snap_create 2026-02-25T08:41:58.196 INFO:tasks.rados.rados.0.trial039.stdout:93: finishing write tid 2 to trial03942481-42 2026-02-25T08:41:58.196 INFO:tasks.rados.rados.0.trial039.stdout:93: finishing write tid 3 to trial03942481-42 2026-02-25T08:41:58.196 INFO:tasks.rados.rados.0.trial039.stdout:93: finishing write tid 4 to trial03942481-42 2026-02-25T08:41:58.197 INFO:tasks.rados.rados.0.trial039.stdout:93: oid 42 updating version 0 to 34 2026-02-25T08:41:58.197 INFO:tasks.rados.rados.0.trial039.stdout:93: oid 42 version 34 is already newer than 34 2026-02-25T08:41:58.197 INFO:tasks.rados.rados.0.trial039.stdout:93: oid 42 version 34 is already newer than 34 2026-02-25T08:41:58.197 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 42 v 34 (ObjNum 64 snap 4 seq_num 64) dirty exists 2026-02-25T08:41:58.197 INFO:tasks.rados.rados.0.trial039.stdout:93: left oid 42 (ObjNum 64 snap 4 seq_num 64) 2026-02-25T08:41:58.237 INFO:tasks.rados.rados.0.trial039.stdout:96: finishing write tid 1 to trial03942481-41 2026-02-25T08:41:58.238 INFO:tasks.rados.rados.0.trial039.stdout:97: finishing write tid 1 to trial03942481-35 2026-02-25T08:41:58.240 INFO:tasks.rados.rados.0.trial039.stdout:96: finishing write tid 2 to trial03942481-41 2026-02-25T08:41:58.245 INFO:tasks.rados.rados.0.trial039.stdout:96: finishing write tid 3 to trial03942481-41 2026-02-25T08:41:58.248 INFO:tasks.rados.rados.0.trial039.stdout:96: finishing write tid 4 to trial03942481-41 2026-02-25T08:41:58.248 INFO:tasks.rados.rados.0.trial039.stdout:96: oid 41 updating version 0 to 15 2026-02-25T08:41:58.248 INFO:tasks.rados.rados.0.trial039.stdout:96: oid 41 updating version 15 to 17 2026-02-25T08:41:58.249 INFO:tasks.rados.rados.0.trial039.stdout:96: oid 41 version 17 is already newer than 17 2026-02-25T08:41:58.249 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 41 v 17 (ObjNum 65 snap 4 seq_num 65) dirty exists 2026-02-25T08:41:58.249 INFO:tasks.rados.rados.0.trial039.stdout:96: left oid 41 (ObjNum 65 snap 4 seq_num 65) 2026-02-25T08:41:58.249 INFO:tasks.rados.rados.0.trial039.stdout:97: finishing write tid 2 to trial03942481-35 2026-02-25T08:41:58.261 INFO:tasks.rados.rados.0.trial039.stdout:97: finishing write tid 3 to trial03942481-35 2026-02-25T08:41:58.268 INFO:tasks.rados.rados.0.trial039.stdout:97: finishing write tid 4 to trial03942481-35 2026-02-25T08:41:58.270 INFO:tasks.rados.rados.0.trial039.stdout:97: finishing write tid 5 to trial03942481-35 2026-02-25T08:41:58.270 INFO:tasks.rados.rados.0.trial039.stdout:97: oid 35 updating version 0 to 10 2026-02-25T08:41:58.270 INFO:tasks.rados.rados.0.trial039.stdout:97: oid 35 version 10 is already newer than 10 2026-02-25T08:41:58.270 INFO:tasks.rados.rados.0.trial039.stdout:97: oid 35 version 10 is already newer than 10 2026-02-25T08:41:58.271 INFO:tasks.rados.rados.0.trial039.stdout:97: oid 35 version 10 is already newer than 10 2026-02-25T08:41:58.271 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 35 v 10 (ObjNum 66 snap 4 seq_num 66) dirty exists 2026-02-25T08:41:58.271 INFO:tasks.rados.rados.0.trial039.stdout:97: left oid 35 (ObjNum 66 snap 4 seq_num 66) 2026-02-25T08:41:59.137 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:41:59.181 INFO:tasks.rados.rados.0.trial039.stdout:93: done (3 left) 2026-02-25T08:41:59.181 INFO:tasks.rados.rados.0.trial039.stdout:96: done (2 left) 2026-02-25T08:41:59.181 INFO:tasks.rados.rados.0.trial039.stdout:97: done (1 left) 2026-02-25T08:41:59.182 INFO:tasks.rados.rados.0.trial039.stdout:98: done (0 left) 2026-02-25T08:41:59.182 INFO:tasks.rados.rados.0.trial039.stdout:99: write (excl) oid 42 current snap is 5 2026-02-25T08:41:59.182 INFO:tasks.rados.rados.0.trial039.stdout:99: seq_num 67 ranges {726069=772507,2071117=715676} 2026-02-25T08:41:59.186 INFO:tasks.rados.rados.0.trial039.stdout:99: writing trial03942481-42 from 726069 to 1498576 tid 1 2026-02-25T08:41:59.192 INFO:tasks.rados.rados.0.trial039.stdout:99: writing trial03942481-42 from 2071117 to 2786793 tid 2 2026-02-25T08:41:59.192 INFO:tasks.rados.rados.0.trial039.stdout:100: snap_remove snap 4 2026-02-25T08:41:59.212 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:59.216 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:41:59.691 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:59.767 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:59.764+0000 7f166e19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:59.770 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:41:59.768+0000 7f166e19c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:59.796 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:41:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:41:59.812 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:00.183 INFO:tasks.rados.rados.0.trial039.stdout:100: done (1 left) 2026-02-25T08:42:00.183 INFO:tasks.rados.rados.0.trial039.stdout:101: read oid 2 snap -1 2026-02-25T08:42:00.183 INFO:tasks.rados.rados.0.trial039.stdout:101: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:42:00.183 INFO:tasks.rados.rados.0.trial039.stdout:102: read oid 49 snap -1 2026-02-25T08:42:00.183 INFO:tasks.rados.rados.0.trial039.stdout:102: expect (ObjNum 58 snap 1 seq_num 58) 2026-02-25T08:42:00.184 INFO:tasks.rados.rados.0.trial039.stdout:103: read oid 38 snap -1 2026-02-25T08:42:00.184 INFO:tasks.rados.rados.0.trial039.stdout:103: expect (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:42:00.184 INFO:tasks.rados.rados.0.trial039.stdout:104: write oid 17 current snap is 5 2026-02-25T08:42:00.184 INFO:tasks.rados.rados.0.trial039.stdout:99: finishing write tid 1 to trial03942481-42 2026-02-25T08:42:00.184 INFO:tasks.rados.rados.0.trial039.stdout:104: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2026-02-25T08:42:00.186 INFO:tasks.rados.rados.0.trial039.stdout:104: writing trial03942481-17 from 742876 to 1190632 tid 1 2026-02-25T08:42:00.191 INFO:tasks.rados.rados.0.trial039.stdout:104: writing trial03942481-17 from 1729599 to 2305286 tid 2 2026-02-25T08:42:00.195 INFO:tasks.rados.rados.0.trial039.stdout:104: writing trial03942481-17 from 3101234 to 3142876 tid 3 2026-02-25T08:42:00.195 INFO:tasks.rados.rados.0.trial039.stdout:105: read oid 48 snap -1 2026-02-25T08:42:00.195 INFO:tasks.rados.rados.0.trial039.stdout:105: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:42:00.195 INFO:tasks.rados.rados.0.trial039.stdout:106: delete oid 32 current snap is 5 2026-02-25T08:42:00.195 INFO:tasks.rados.rados.0.trial039.stdout:99: finishing write tid 2 to trial03942481-42 2026-02-25T08:42:00.196 INFO:tasks.rados.rados.0.trial039.stdout:99: finishing write tid 3 to trial03942481-42 2026-02-25T08:42:00.196 INFO:tasks.rados.rados.0.trial039.stdout:99: finishing write tid 4 to trial03942481-42 2026-02-25T08:42:00.196 INFO:tasks.rados.rados.0.trial039.stdout:99: oid 42 updating version 0 to 42 2026-02-25T08:42:00.196 INFO:tasks.rados.rados.0.trial039.stdout:99: oid 42 version 42 is already newer than 40 2026-02-25T08:42:00.196 INFO:tasks.rados.rados.0.trial039.stdout:99: oid 42 version 42 is already newer than 42 2026-02-25T08:42:00.196 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 42 v 42 (ObjNum 67 snap 5 seq_num 67) dirty exists 2026-02-25T08:42:00.196 INFO:tasks.rados.rados.0.trial039.stdout:99: left oid 42 (ObjNum 67 snap 5 seq_num 67) 2026-02-25T08:42:00.206 INFO:tasks.rados.rados.0.trial039.stdout:105: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:42:00.209 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1594103 2026-02-25T08:42:00.262 INFO:tasks.rados.rados.0.trial039.stdout:103: expect (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:42:00.317 INFO:tasks.rados.rados.0.trial039.stdout:102: expect (ObjNum 58 snap 1 seq_num 58) 2026-02-25T08:42:00.408 INFO:tasks.rados.rados.0.trial039.stdout:101: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:42:00.409 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:00.446 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1469399 2026-02-25T08:42:00.479 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2475839 2026-02-25T08:42:00.483 INFO:tasks.rados.rados.0.trial039.stdout:104: finishing write tid 1 to trial03942481-17 2026-02-25T08:42:00.483 INFO:tasks.rados.rados.0.trial039.stdout:104: finishing write tid 2 to trial03942481-17 2026-02-25T08:42:00.483 INFO:tasks.rados.rados.0.trial039.stdout:99: done (6 left) 2026-02-25T08:42:00.483 INFO:tasks.rados.rados.0.trial039.stdout:101: done (5 left) 2026-02-25T08:42:00.483 INFO:tasks.rados.rados.0.trial039.stdout:102: done (4 left) 2026-02-25T08:42:00.483 INFO:tasks.rados.rados.0.trial039.stdout:103: done (3 left) 2026-02-25T08:42:00.483 INFO:tasks.rados.rados.0.trial039.stdout:105: done (2 left) 2026-02-25T08:42:00.484 INFO:tasks.rados.rados.0.trial039.stdout:106: done (1 left) 2026-02-25T08:42:00.484 INFO:tasks.rados.rados.0.trial039.stdout:107: write (excl) oid 49 current snap is 5 2026-02-25T08:42:00.484 INFO:tasks.rados.rados.0.trial039.stdout:104: finishing write tid 3 to trial03942481-17 2026-02-25T08:42:00.484 INFO:tasks.rados.rados.0.trial039.stdout:107: seq_num 70 ranges {776490=714607,2246563=579356} 2026-02-25T08:42:00.488 INFO:tasks.rados.rados.0.trial039.stdout:107: writing trial03942481-49 from 776490 to 1491097 tid 1 2026-02-25T08:42:00.494 INFO:tasks.rados.rados.0.trial039.stdout:107: writing trial03942481-49 from 2246563 to 2825919 tid 2 2026-02-25T08:42:00.494 INFO:tasks.rados.rados.0.trial039.stdout:108: delete oid 13 current snap is 5 2026-02-25T08:42:00.494 INFO:tasks.rados.rados.0.trial039.stdout:104: finishing write tid 4 to trial03942481-17 2026-02-25T08:42:00.494 INFO:tasks.rados.rados.0.trial039.stdout:104: finishing write tid 5 to trial03942481-17 2026-02-25T08:42:00.494 INFO:tasks.rados.rados.0.trial039.stdout:104: oid 17 updating version 0 to 28 2026-02-25T08:42:00.494 INFO:tasks.rados.rados.0.trial039.stdout:104: oid 17 version 28 is already newer than 28 2026-02-25T08:42:00.494 INFO:tasks.rados.rados.0.trial039.stdout:104: oid 17 version 28 is already newer than 28 2026-02-25T08:42:00.494 INFO:tasks.rados.rados.0.trial039.stdout:104: oid 17 version 28 is already newer than 28 2026-02-25T08:42:00.494 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 17 v 28 (ObjNum 68 snap 5 seq_num 68) dirty exists 2026-02-25T08:42:00.494 INFO:tasks.rados.rados.0.trial039.stdout:104: left oid 17 (ObjNum 68 snap 5 seq_num 68) 2026-02-25T08:42:00.507 INFO:tasks.rados.rados.0.trial039.stdout:107: finishing write tid 1 to trial03942481-49 2026-02-25T08:42:00.507 INFO:tasks.rados.rados.0.trial039.stdout:104: done (2 left) 2026-02-25T08:42:00.507 INFO:tasks.rados.rados.0.trial039.stdout:108: done (1 left) 2026-02-25T08:42:00.507 INFO:tasks.rados.rados.0.trial039.stdout:109: write (excl) oid 12 current snap is 5 2026-02-25T08:42:00.507 INFO:tasks.rados.rados.0.trial039.stdout:109: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2026-02-25T08:42:00.510 INFO:tasks.rados.rados.0.trial039.stdout:109: writing trial03942481-12 from 410104 to 875209 tid 1 2026-02-25T08:42:00.515 INFO:tasks.rados.rados.0.trial039.stdout:109: writing trial03942481-12 from 1563527 to 2146552 tid 2 2026-02-25T08:42:00.519 INFO:tasks.rados.rados.0.trial039.stdout:109: writing trial03942481-12 from 2810926 to 3210104 tid 3 2026-02-25T08:42:00.519 INFO:tasks.rados.rados.0.trial039.stdout:110: read oid 44 snap -1 2026-02-25T08:42:00.519 INFO:tasks.rados.rados.0.trial039.stdout:110: expect (ObjNum 54 snap 1 seq_num 54) 2026-02-25T08:42:00.519 INFO:tasks.rados.rados.0.trial039.stdout:111: delete oid 27 current snap is 5 2026-02-25T08:42:00.524 INFO:tasks.rados.rados.0.trial039.stdout:107: finishing write tid 2 to trial03942481-49 2026-02-25T08:42:00.528 INFO:tasks.rados.rados.0.trial039.stdout:109: finishing write tid 1 to trial03942481-12 2026-02-25T08:42:00.536 INFO:tasks.rados.rados.0.trial039.stdout:107: finishing write tid 3 to trial03942481-49 2026-02-25T08:42:00.538 INFO:tasks.rados.rados.0.trial039.stdout:107: finishing write tid 4 to trial03942481-49 2026-02-25T08:42:00.538 INFO:tasks.rados.rados.0.trial039.stdout:107: oid 49 updating version 0 to 22 2026-02-25T08:42:00.538 INFO:tasks.rados.rados.0.trial039.stdout:107: oid 49 version 22 is already newer than 22 2026-02-25T08:42:00.538 INFO:tasks.rados.rados.0.trial039.stdout:107: oid 49 version 22 is already newer than 20 2026-02-25T08:42:00.538 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 49 v 22 (ObjNum 70 snap 5 seq_num 70) dirty exists 2026-02-25T08:42:00.538 INFO:tasks.rados.rados.0.trial039.stdout:107: left oid 49 (ObjNum 70 snap 5 seq_num 70) 2026-02-25T08:42:00.544 INFO:tasks.rados.rados.0.trial039.stdout:109: finishing write tid 2 to trial03942481-12 2026-02-25T08:42:00.549 INFO:tasks.rados.rados.0.trial039.stdout:107: done (3 left) 2026-02-25T08:42:00.549 INFO:tasks.rados.rados.0.trial039.stdout:111: done (2 left) 2026-02-25T08:42:00.550 INFO:tasks.rados.rados.0.trial039.stdout:112: snap_create 2026-02-25T08:42:00.550 INFO:tasks.rados.rados.0.trial039.stdout:109: finishing write tid 3 to trial03942481-12 2026-02-25T08:42:00.553 INFO:tasks.rados.rados.0.trial039.stdout:109: finishing write tid 4 to trial03942481-12 2026-02-25T08:42:00.555 INFO:tasks.rados.rados.0.trial039.stdout:109: finishing write tid 5 to trial03942481-12 2026-02-25T08:42:00.555 INFO:tasks.rados.rados.0.trial039.stdout:109: oid 12 updating version 0 to 10 2026-02-25T08:42:00.555 INFO:tasks.rados.rados.0.trial039.stdout:109: oid 12 version 10 is already newer than 7 2026-02-25T08:42:00.555 INFO:tasks.rados.rados.0.trial039.stdout:109: oid 12 version 10 is already newer than 10 2026-02-25T08:42:00.555 INFO:tasks.rados.rados.0.trial039.stdout:109: oid 12 version 10 is already newer than 10 2026-02-25T08:42:00.555 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 12 v 10 (ObjNum 72 snap 5 seq_num 72) dirty exists 2026-02-25T08:42:00.555 INFO:tasks.rados.rados.0.trial039.stdout:109: left oid 12 (ObjNum 72 snap 5 seq_num 72) 2026-02-25T08:42:00.557 INFO:tasks.rados.rados.0.trial039.stdout:110: expect (ObjNum 54 snap 1 seq_num 54) 2026-02-25T08:42:01.186 INFO:tasks.rados.rados.0.trial039.stdout:109: done (2 left) 2026-02-25T08:42:01.187 INFO:tasks.rados.rados.0.trial039.stdout:110: done (1 left) 2026-02-25T08:42:01.187 INFO:tasks.rados.rados.0.trial039.stdout:112: done (0 left) 2026-02-25T08:42:01.187 INFO:tasks.rados.rados.0.trial039.stdout:113: delete oid 48 current snap is 6 2026-02-25T08:42:01.218 INFO:tasks.rados.rados.0.trial039.stdout:113: done (0 left) 2026-02-25T08:42:01.218 INFO:tasks.rados.rados.0.trial039.stdout:114: read oid 45 snap 5 2026-02-25T08:42:01.218 INFO:tasks.rados.rados.0.trial039.stdout:114: expect (ObjNum 53 snap 1 seq_num 53) 2026-02-25T08:42:01.218 INFO:tasks.rados.rados.0.trial039.stdout:115: snap_create 2026-02-25T08:42:01.238 INFO:tasks.rados.rados.0.trial039.stdout:114: expect (ObjNum 53 snap 1 seq_num 53) 2026-02-25T08:42:01.312 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1632643 2026-02-25T08:42:01.814 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:01.892 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:01.890+0000 7f4ce7793640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:01.894 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:01.893+0000 7f4ce7793640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:01.920 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:01.936 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:02.191 INFO:tasks.rados.rados.0.trial039.stdout:114: done (1 left) 2026-02-25T08:42:02.191 INFO:tasks.rados.rados.0.trial039.stdout:115: done (0 left) 2026-02-25T08:42:02.191 INFO:tasks.rados.rados.0.trial039.stdout:116: snap_create 2026-02-25T08:42:02.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~28s 2026-02-25T08:42:03.195 INFO:tasks.rados.rados.0.trial039.stdout:116: done (0 left) 2026-02-25T08:42:03.195 INFO:tasks.rados.rados.0.trial039.stdout:117: snap_remove snap 6 2026-02-25T08:42:03.938 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:04.015 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:04.011+0000 7f0d69fed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:04.017 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:04.015+0000 7f0d69fed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:04.043 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:04.059 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:04.200 INFO:tasks.rados.rados.0.trial039.stdout:117: done (0 left) 2026-02-25T08:42:04.200 INFO:tasks.rados.rados.0.trial039.stdout:118: write (excl) oid 26 current snap is 8 2026-02-25T08:42:04.200 INFO:tasks.rados.rados.0.trial039.stdout:118: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2026-02-25T08:42:04.204 INFO:tasks.rados.rados.0.trial039.stdout:118: writing trial03942481-26 from 460525 to 1151377 tid 1 2026-02-25T08:42:04.209 INFO:tasks.rados.rados.0.trial039.stdout:118: writing trial03942481-26 from 1622620 to 2352972 tid 2 2026-02-25T08:42:04.213 INFO:tasks.rados.rados.0.trial039.stdout:118: writing trial03942481-26 from 3060489 to 3260525 tid 3 2026-02-25T08:42:04.213 INFO:tasks.rados.rados.0.trial039.stdout:119: snap_remove snap 5 2026-02-25T08:42:04.217 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:04.286 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:04.290 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:05.202 INFO:tasks.rados.rados.0.trial039.stdout:119: done (1 left) 2026-02-25T08:42:05.203 INFO:tasks.rados.rados.0.trial039.stdout:120: read oid 7 snap -1 2026-02-25T08:42:05.203 INFO:tasks.rados.rados.0.trial039.stdout:120: expect (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:42:05.203 INFO:tasks.rados.rados.0.trial039.stdout:121: delete oid 3 current snap is 8 2026-02-25T08:42:05.203 INFO:tasks.rados.rados.0.trial039.stdout:118: finishing write tid 1 to trial03942481-26 2026-02-25T08:42:05.203 INFO:tasks.rados.rados.0.trial039.stdout:118: finishing write tid 2 to trial03942481-26 2026-02-25T08:42:05.203 INFO:tasks.rados.rados.0.trial039.stdout:118: finishing write tid 3 to trial03942481-26 2026-02-25T08:42:05.203 INFO:tasks.rados.rados.0.trial039.stdout:118: finishing write tid 4 to trial03942481-26 2026-02-25T08:42:05.203 INFO:tasks.rados.rados.0.trial039.stdout:118: finishing write tid 5 to trial03942481-26 2026-02-25T08:42:05.203 INFO:tasks.rados.rados.0.trial039.stdout:118: oid 26 updating version 0 to 15 2026-02-25T08:42:05.203 INFO:tasks.rados.rados.0.trial039.stdout:118: oid 26 updating version 15 to 18 2026-02-25T08:42:05.204 INFO:tasks.rados.rados.0.trial039.stdout:118: oid 26 version 18 is already newer than 18 2026-02-25T08:42:05.204 INFO:tasks.rados.rados.0.trial039.stdout:118: oid 26 version 18 is already newer than 18 2026-02-25T08:42:05.204 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 26 v 18 (ObjNum 75 snap 8 seq_num 75) dirty exists 2026-02-25T08:42:05.204 INFO:tasks.rados.rados.0.trial039.stdout:118: left oid 26 (ObjNum 75 snap 8 seq_num 75) 2026-02-25T08:42:05.232 INFO:tasks.rados.rados.0.trial039.stdout:120: expect (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:42:05.234 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1631716 2026-02-25T08:42:05.242 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:05.278 INFO:tasks.rados.rados.0.trial039.stdout:118: done (2 left) 2026-02-25T08:42:05.278 INFO:tasks.rados.rados.0.trial039.stdout:120: done (1 left) 2026-02-25T08:42:05.278 INFO:tasks.rados.rados.0.trial039.stdout:121: done (0 left) 2026-02-25T08:42:05.278 INFO:tasks.rados.rados.0.trial039.stdout:122: snap_remove snap 7 2026-02-25T08:42:06.060 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:06.136 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:06.132+0000 7f3dc390b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:06.139 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:06.137+0000 7f3dc390b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:06.164 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:06.180 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:06.206 INFO:tasks.rados.rados.0.trial039.stdout:122: done (0 left) 2026-02-25T08:42:06.206 INFO:tasks.rados.rados.0.trial039.stdout:123: write (excl) oid 16 current snap is 8 2026-02-25T08:42:06.206 INFO:tasks.rados.rados.0.trial039.stdout:123: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2026-02-25T08:42:06.210 INFO:tasks.rados.rados.0.trial039.stdout:123: writing trial03942481-16 from 494139 to 1051842 tid 1 2026-02-25T08:42:06.214 INFO:tasks.rados.rados.0.trial039.stdout:123: writing trial03942481-16 from 1455937 to 2189958 tid 2 2026-02-25T08:42:06.219 INFO:tasks.rados.rados.0.trial039.stdout:123: writing trial03942481-16 from 2948041 to 3294139 tid 3 2026-02-25T08:42:06.220 INFO:tasks.rados.rados.0.trial039.stdout:124: write (excl) oid 25 current snap is 8 2026-02-25T08:42:06.220 INFO:tasks.rados.rados.0.trial039.stdout:124: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2026-02-25T08:42:06.223 INFO:tasks.rados.rados.0.trial039.stdout:124: writing trial03942481-25 from 510946 to 1143898 tid 1 2026-02-25T08:42:06.227 INFO:tasks.rados.rados.0.trial039.stdout:124: writing trial03942481-25 from 1798066 to 2275745 tid 2 2026-02-25T08:42:06.231 INFO:tasks.rados.rados.0.trial039.stdout:124: writing trial03942481-25 from 2742758 to 3199975 tid 3 2026-02-25T08:42:06.231 INFO:tasks.rados.rados.0.trial039.stdout:125: write (excl) oid 46 current snap is 8 2026-02-25T08:42:06.232 INFO:tasks.rados.rados.0.trial039.stdout:125: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2026-02-25T08:42:06.236 INFO:tasks.rados.rados.0.trial039.stdout:125: writing trial03942481-46 from 527753 to 1235954 tid 1 2026-02-25T08:42:06.241 INFO:tasks.rados.rados.0.trial039.stdout:125: writing trial03942481-46 from 1856548 to 2594238 tid 2 2026-02-25T08:42:06.244 INFO:tasks.rados.rados.0.trial039.stdout:125: writing trial03942481-46 from 3286534 to 3327753 tid 3 2026-02-25T08:42:06.244 INFO:tasks.rados.rados.0.trial039.stdout:126: read oid 43 snap -1 2026-02-25T08:42:06.244 INFO:tasks.rados.rados.0.trial039.stdout:126: expect deleted 2026-02-25T08:42:06.244 INFO:tasks.rados.rados.0.trial039.stdout:127: write (excl) oid 47 current snap is 8 2026-02-25T08:42:06.244 INFO:tasks.rados.rados.0.trial039.stdout:127: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2026-02-25T08:42:06.249 INFO:tasks.rados.rados.0.trial039.stdout:127: writing trial03942481-47 from 544560 to 1328010 tid 1 2026-02-25T08:42:06.253 INFO:tasks.rados.rados.0.trial039.stdout:127: writing trial03942481-47 from 1915030 to 2396378 tid 2 2026-02-25T08:42:06.257 INFO:tasks.rados.rados.0.trial039.stdout:127: writing trial03942481-47 from 2797604 to 3277365 tid 3 2026-02-25T08:42:06.257 INFO:tasks.rados.rados.0.trial039.stdout:128: delete oid 9 current snap is 8 2026-02-25T08:42:06.257 INFO:tasks.rados.rados.0.trial039.stdout:123: finishing write tid 1 to trial03942481-16 2026-02-25T08:42:06.271 INFO:tasks.rados.rados.0.trial039.stdout:126: done (5 left) 2026-02-25T08:42:06.272 INFO:tasks.rados.rados.0.trial039.stdout:128: done (4 left) 2026-02-25T08:42:06.272 INFO:tasks.rados.rados.0.trial039.stdout:129: snap_create 2026-02-25T08:42:06.274 INFO:tasks.rados.rados.0.trial039.stdout:125: finishing write tid 1 to trial03942481-46 2026-02-25T08:42:06.274 INFO:tasks.rados.rados.0.trial039.stdout:124: finishing write tid 1 to trial03942481-25 2026-02-25T08:42:06.284 INFO:tasks.rados.rados.0.trial039.stdout:123: finishing write tid 2 to trial03942481-16 2026-02-25T08:42:06.293 INFO:tasks.rados.rados.0.trial039.stdout:127: finishing write tid 1 to trial03942481-47 2026-02-25T08:42:06.303 INFO:tasks.rados.rados.0.trial039.stdout:124: finishing write tid 2 to trial03942481-25 2026-02-25T08:42:06.312 INFO:tasks.rados.rados.0.trial039.stdout:125: finishing write tid 2 to trial03942481-46 2026-02-25T08:42:06.317 INFO:tasks.rados.rados.0.trial039.stdout:124: finishing write tid 3 to trial03942481-25 2026-02-25T08:42:06.324 INFO:tasks.rados.rados.0.trial039.stdout:125: finishing write tid 3 to trial03942481-46 2026-02-25T08:42:06.326 INFO:tasks.rados.rados.0.trial039.stdout:124: finishing write tid 4 to trial03942481-25 2026-02-25T08:42:06.327 INFO:tasks.rados.rados.0.trial039.stdout:124: finishing write tid 5 to trial03942481-25 2026-02-25T08:42:06.327 INFO:tasks.rados.rados.0.trial039.stdout:124: oid 25 updating version 0 to 7 2026-02-25T08:42:06.327 INFO:tasks.rados.rados.0.trial039.stdout:124: oid 25 version 7 is already newer than 7 2026-02-25T08:42:06.327 INFO:tasks.rados.rados.0.trial039.stdout:124: oid 25 version 7 is already newer than 7 2026-02-25T08:42:06.327 INFO:tasks.rados.rados.0.trial039.stdout:124: oid 25 version 7 is already newer than 4 2026-02-25T08:42:06.327 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 25 v 7 (ObjNum 78 snap 8 seq_num 78) dirty exists 2026-02-25T08:42:06.328 INFO:tasks.rados.rados.0.trial039.stdout:124: left oid 25 (ObjNum 78 snap 8 seq_num 78) 2026-02-25T08:42:06.332 INFO:tasks.rados.rados.0.trial039.stdout:123: finishing write tid 3 to trial03942481-16 2026-02-25T08:42:06.333 INFO:tasks.rados.rados.0.trial039.stdout:125: finishing write tid 4 to trial03942481-46 2026-02-25T08:42:06.335 INFO:tasks.rados.rados.0.trial039.stdout:125: finishing write tid 5 to trial03942481-46 2026-02-25T08:42:06.335 INFO:tasks.rados.rados.0.trial039.stdout:125: oid 46 updating version 0 to 14 2026-02-25T08:42:06.335 INFO:tasks.rados.rados.0.trial039.stdout:125: oid 46 updating version 14 to 17 2026-02-25T08:42:06.335 INFO:tasks.rados.rados.0.trial039.stdout:125: oid 46 version 17 is already newer than 17 2026-02-25T08:42:06.335 INFO:tasks.rados.rados.0.trial039.stdout:125: oid 46 version 17 is already newer than 17 2026-02-25T08:42:06.336 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 46 v 17 (ObjNum 79 snap 8 seq_num 79) dirty exists 2026-02-25T08:42:06.336 INFO:tasks.rados.rados.0.trial039.stdout:125: left oid 46 (ObjNum 79 snap 8 seq_num 79) 2026-02-25T08:42:06.339 INFO:tasks.rados.rados.0.trial039.stdout:123: finishing write tid 4 to trial03942481-16 2026-02-25T08:42:06.340 INFO:tasks.rados.rados.0.trial039.stdout:127: finishing write tid 2 to trial03942481-47 2026-02-25T08:42:06.341 INFO:tasks.rados.rados.0.trial039.stdout:123: finishing write tid 5 to trial03942481-16 2026-02-25T08:42:06.341 INFO:tasks.rados.rados.0.trial039.stdout:123: oid 16 updating version 0 to 14 2026-02-25T08:42:06.342 INFO:tasks.rados.rados.0.trial039.stdout:123: oid 16 version 14 is already newer than 14 2026-02-25T08:42:06.342 INFO:tasks.rados.rados.0.trial039.stdout:123: oid 16 version 14 is already newer than 14 2026-02-25T08:42:06.342 INFO:tasks.rados.rados.0.trial039.stdout:123: oid 16 version 14 is already newer than 11 2026-02-25T08:42:06.342 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 16 v 14 (ObjNum 77 snap 8 seq_num 77) dirty exists 2026-02-25T08:42:06.342 INFO:tasks.rados.rados.0.trial039.stdout:123: left oid 16 (ObjNum 77 snap 8 seq_num 77) 2026-02-25T08:42:06.346 INFO:tasks.rados.rados.0.trial039.stdout:127: finishing write tid 3 to trial03942481-47 2026-02-25T08:42:06.351 INFO:tasks.rados.rados.0.trial039.stdout:127: finishing write tid 4 to trial03942481-47 2026-02-25T08:42:06.352 INFO:tasks.rados.rados.0.trial039.stdout:127: finishing write tid 5 to trial03942481-47 2026-02-25T08:42:06.353 INFO:tasks.rados.rados.0.trial039.stdout:127: oid 47 updating version 0 to 14 2026-02-25T08:42:06.353 INFO:tasks.rados.rados.0.trial039.stdout:127: oid 47 version 14 is already newer than 11 2026-02-25T08:42:06.353 INFO:tasks.rados.rados.0.trial039.stdout:127: oid 47 version 14 is already newer than 14 2026-02-25T08:42:06.353 INFO:tasks.rados.rados.0.trial039.stdout:127: oid 47 version 14 is already newer than 14 2026-02-25T08:42:06.353 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 47 v 14 (ObjNum 80 snap 8 seq_num 80) dirty exists 2026-02-25T08:42:06.353 INFO:tasks.rados.rados.0.trial039.stdout:127: left oid 47 (ObjNum 80 snap 8 seq_num 80) 2026-02-25T08:42:07.211 INFO:tasks.rados.rados.0.trial039.stdout:123: done (4 left) 2026-02-25T08:42:07.211 INFO:tasks.rados.rados.0.trial039.stdout:124: done (3 left) 2026-02-25T08:42:07.212 INFO:tasks.rados.rados.0.trial039.stdout:125: done (2 left) 2026-02-25T08:42:07.212 INFO:tasks.rados.rados.0.trial039.stdout:127: done (1 left) 2026-02-25T08:42:07.212 INFO:tasks.rados.rados.0.trial039.stdout:129: done (0 left) 2026-02-25T08:42:07.212 INFO:tasks.rados.rados.0.trial039.stdout:130: read oid 1 snap -1 2026-02-25T08:42:07.212 INFO:tasks.rados.rados.0.trial039.stdout:130: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:42:07.212 INFO:tasks.rados.rados.0.trial039.stdout:131: write (excl) oid 11 current snap is 9 2026-02-25T08:42:07.212 INFO:tasks.rados.rados.0.trial039.stdout:131: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2026-02-25T08:42:07.215 INFO:tasks.rados.rados.0.trial039.stdout:131: writing trial03942481-11 from 578174 to 1112122 tid 1 2026-02-25T08:42:07.219 INFO:tasks.rados.rados.0.trial039.stdout:131: writing trial03942481-11 from 1515641 to 2000658 tid 2 2026-02-25T08:42:07.224 INFO:tasks.rados.rados.0.trial039.stdout:131: writing trial03942481-11 from 2736097 to 3238402 tid 3 2026-02-25T08:42:07.224 INFO:tasks.rados.rados.0.trial039.stdout:132: write oid 12 current snap is 9 2026-02-25T08:42:07.224 INFO:tasks.rados.rados.0.trial039.stdout:132: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2026-02-25T08:42:07.228 INFO:tasks.rados.rados.0.trial039.stdout:132: writing trial03942481-12 from 594981 to 1204178 tid 1 2026-02-25T08:42:07.234 INFO:tasks.rados.rados.0.trial039.stdout:132: writing trial03942481-12 from 1974123 to 2719151 tid 2 2026-02-25T08:42:07.237 INFO:tasks.rados.rados.0.trial039.stdout:132: writing trial03942481-12 from 3279873 to 3394981 tid 3 2026-02-25T08:42:07.237 INFO:tasks.rados.rados.0.trial039.stdout:133: write oid 50 current snap is 9 2026-02-25T08:42:07.237 INFO:tasks.rados.rados.0.trial039.stdout:133: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2026-02-25T08:42:07.241 INFO:tasks.rados.rados.0.trial039.stdout:133: writing trial03942481-50 from 611788 to 1012587 tid 1 2026-02-25T08:42:07.245 INFO:tasks.rados.rados.0.trial039.stdout:133: writing trial03942481-50 from 1748958 to 2237644 tid 2 2026-02-25T08:42:07.250 INFO:tasks.rados.rados.0.trial039.stdout:133: writing trial03942481-50 from 2907296 to 3411788 tid 3 2026-02-25T08:42:07.250 INFO:tasks.rados.rados.0.trial039.stdout:134: write oid 15 current snap is 9 2026-02-25T08:42:07.250 INFO:tasks.rados.rados.0.trial039.stdout:134: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2026-02-25T08:42:07.254 INFO:tasks.rados.rados.0.trial039.stdout:134: writing trial03942481-15 from 628595 to 1104643 tid 1 2026-02-25T08:42:07.259 INFO:tasks.rados.rados.0.trial039.stdout:134: writing trial03942481-15 from 1807440 to 2439784 tid 2 2026-02-25T08:42:07.263 INFO:tasks.rados.rados.0.trial039.stdout:134: writing trial03942481-15 from 2934719 to 3387193 tid 3 2026-02-25T08:42:07.263 INFO:tasks.rados.rados.0.trial039.stdout:135: snap_remove snap 8 2026-02-25T08:42:08.181 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:08.211 INFO:tasks.rados.rados.0.trial039.stdout:135: done (5 left) 2026-02-25T08:42:08.211 INFO:tasks.rados.rados.0.trial039.stdout:136: delete oid 8 current snap is 9 2026-02-25T08:42:08.211 INFO:tasks.rados.rados.0.trial039.stdout:130: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:42:08.212 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 328614 2026-02-25T08:42:08.227 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1189448 2026-02-25T08:42:08.237 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:08.255 INFO:tasks.rados.rados.0.trial039.stdout:131: finishing write tid 1 to trial03942481-11 2026-02-25T08:42:08.255 INFO:tasks.rados.rados.0.trial039.stdout:130: done (5 left) 2026-02-25T08:42:08.255 INFO:tasks.rados.rados.0.trial039.stdout:136: done (4 left) 2026-02-25T08:42:08.255 INFO:tasks.rados.rados.0.trial039.stdout:137: write (excl) oid 10 current snap is 9 2026-02-25T08:42:08.255 INFO:tasks.rados.rados.0.trial039.stdout:132: finishing write tid 1 to trial03942481-12 2026-02-25T08:42:08.255 INFO:tasks.rados.rados.0.trial039.stdout:132: finishing write tid 2 to trial03942481-12 2026-02-25T08:42:08.255 INFO:tasks.rados.rados.0.trial039.stdout:131: finishing write tid 2 to trial03942481-11 2026-02-25T08:42:08.255 INFO:tasks.rados.rados.0.trial039.stdout:133: finishing write tid 1 to trial03942481-50 2026-02-25T08:42:08.256 INFO:tasks.rados.rados.0.trial039.stdout:132: finishing write tid 3 to trial03942481-12 2026-02-25T08:42:08.256 INFO:tasks.rados.rados.0.trial039.stdout:137: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2026-02-25T08:42:08.259 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:08.255+0000 7f06690b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:08.260 INFO:tasks.rados.rados.0.trial039.stdout:137: writing trial03942481-10 from 662209 to 1288755 tid 1 2026-02-25T08:42:08.261 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:08.259+0000 7f06690b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:08.264 INFO:tasks.rados.rados.0.trial039.stdout:137: writing trial03942481-10 from 1808051 to 2444064 tid 2 2026-02-25T08:42:08.268 INFO:tasks.rados.rados.0.trial039.stdout:137: writing trial03942481-10 from 2873212 to 3462209 tid 3 2026-02-25T08:42:08.268 INFO:tasks.rados.rados.0.trial039.stdout:138: write (excl) oid 24 current snap is 9 2026-02-25T08:42:08.268 INFO:tasks.rados.rados.0.trial039.stdout:133: finishing write tid 2 to trial03942481-50 2026-02-25T08:42:08.268 INFO:tasks.rados.rados.0.trial039.stdout:138: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2026-02-25T08:42:08.273 INFO:tasks.rados.rados.0.trial039.stdout:138: writing trial03942481-24 from 679016 to 1380811 tid 1 2026-02-25T08:42:08.277 INFO:tasks.rados.rados.0.trial039.stdout:138: writing trial03942481-24 from 1866533 to 2362557 tid 2 2026-02-25T08:42:08.283 INFO:tasks.rados.rados.0.trial039.stdout:138: writing trial03942481-24 from 2900635 to 3303278 tid 3 2026-02-25T08:42:08.283 INFO:tasks.rados.rados.0.trial039.stdout:139: delete oid 16 current snap is 9 2026-02-25T08:42:08.283 INFO:tasks.rados.rados.0.trial039.stdout:131: finishing write tid 3 to trial03942481-11 2026-02-25T08:42:08.283 INFO:tasks.rados.rados.0.trial039.stdout:133: finishing write tid 3 to trial03942481-50 2026-02-25T08:42:08.283 INFO:tasks.rados.rados.0.trial039.stdout:132: finishing write tid 4 to trial03942481-12 2026-02-25T08:42:08.283 INFO:tasks.rados.rados.0.trial039.stdout:133: finishing write tid 4 to trial03942481-50 2026-02-25T08:42:08.283 INFO:tasks.rados.rados.0.trial039.stdout:131: finishing write tid 4 to trial03942481-11 2026-02-25T08:42:08.283 INFO:tasks.rados.rados.0.trial039.stdout:132: finishing write tid 5 to trial03942481-12 2026-02-25T08:42:08.283 INFO:tasks.rados.rados.0.trial039.stdout:132: oid 12 updating version 0 to 19 2026-02-25T08:42:08.283 INFO:tasks.rados.rados.0.trial039.stdout:132: oid 12 version 19 is already newer than 19 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:132: oid 12 version 19 is already newer than 19 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:132: oid 12 version 19 is already newer than 19 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 12 v 19 (ObjNum 83 snap 9 seq_num 83) dirty exists 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:132: left oid 12 (ObjNum 83 snap 9 seq_num 83) 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:133: finishing write tid 5 to trial03942481-50 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:133: oid 50 updating version 0 to 35 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:133: oid 50 version 35 is already newer than 35 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:133: oid 50 version 35 is already newer than 35 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:133: oid 50 version 35 is already newer than 35 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 50 v 35 (ObjNum 84 snap 9 seq_num 84) dirty exists 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:133: left oid 50 (ObjNum 84 snap 9 seq_num 84) 2026-02-25T08:42:08.284 INFO:tasks.rados.rados.0.trial039.stdout:131: finishing write tid 5 to trial03942481-11 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:131: oid 11 updating version 0 to 5 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:131: oid 11 updating version 5 to 8 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:131: oid 11 version 8 is already newer than 8 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:131: oid 11 version 8 is already newer than 8 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 11 v 8 (ObjNum 82 snap 9 seq_num 82) dirty exists 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:131: left oid 11 (ObjNum 82 snap 9 seq_num 82) 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:134: finishing write tid 1 to trial03942481-15 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:134: finishing write tid 2 to trial03942481-15 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:134: finishing write tid 3 to trial03942481-15 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:134: finishing write tid 4 to trial03942481-15 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:134: finishing write tid 5 to trial03942481-15 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:134: oid 15 updating version 0 to 21 2026-02-25T08:42:08.285 INFO:tasks.rados.rados.0.trial039.stdout:134: oid 15 version 21 is already newer than 21 2026-02-25T08:42:08.286 INFO:tasks.rados.rados.0.trial039.stdout:134: oid 15 version 21 is already newer than 21 2026-02-25T08:42:08.286 INFO:tasks.rados.rados.0.trial039.stdout:134: oid 15 version 21 is already newer than 21 2026-02-25T08:42:08.286 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 15 v 21 (ObjNum 85 snap 9 seq_num 85) dirty exists 2026-02-25T08:42:08.286 INFO:tasks.rados.rados.0.trial039.stdout:134: left oid 15 (ObjNum 85 snap 9 seq_num 85) 2026-02-25T08:42:08.286 INFO:tasks.rados.rados.0.trial039.stdout:137: finishing write tid 1 to trial03942481-10 2026-02-25T08:42:08.296 INFO:tasks.rados.rados.0.trial039.stdout:131: done (6 left) 2026-02-25T08:42:08.296 INFO:tasks.rados.rados.0.trial039.stdout:132: done (5 left) 2026-02-25T08:42:08.296 INFO:tasks.rados.rados.0.trial039.stdout:133: done (4 left) 2026-02-25T08:42:08.296 INFO:tasks.rados.rados.0.trial039.stdout:134: done (3 left) 2026-02-25T08:42:08.296 INFO:tasks.rados.rados.0.trial039.stdout:139: done (2 left) 2026-02-25T08:42:08.296 INFO:tasks.rados.rados.0.trial039.stdout:140: write (excl) oid 31 current snap is 9 2026-02-25T08:42:08.296 INFO:tasks.rados.rados.0.trial039.stdout:140: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2026-02-25T08:42:08.300 INFO:tasks.rados.rados.0.trial039.stdout:140: writing trial03942481-31 from 712630 to 1164923 tid 1 2026-02-25T08:42:08.304 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:08.304 INFO:tasks.rados.rados.0.trial039.stdout:140: writing trial03942481-31 from 1583497 to 2083190 tid 2 2026-02-25T08:42:08.308 INFO:tasks.rados.rados.0.trial039.stdout:140: writing trial03942481-31 from 2555481 to 2980668 tid 3 2026-02-25T08:42:08.309 INFO:tasks.rados.rados.0.trial039.stdout:141: write oid 28 current snap is 9 2026-02-25T08:42:08.309 INFO:tasks.rados.rados.0.trial039.stdout:137: finishing write tid 2 to trial03942481-10 2026-02-25T08:42:08.309 INFO:tasks.rados.rados.0.trial039.stdout:141: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2026-02-25T08:42:08.313 INFO:tasks.rados.rados.0.trial039.stdout:141: writing trial03942481-28 from 729437 to 1256979 tid 1 2026-02-25T08:42:08.318 INFO:tasks.rados.rados.0.trial039.stdout:141: writing trial03942481-28 from 1925626 to 2568977 tid 2 2026-02-25T08:42:08.322 INFO:tasks.rados.rados.0.trial039.stdout:141: writing trial03942481-28 from 3266551 to 3529437 tid 3 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:142: snap_create 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:138: finishing write tid 1 to trial03942481-24 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:137: finishing write tid 3 to trial03942481-10 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:137: finishing write tid 4 to trial03942481-10 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:137: finishing write tid 5 to trial03942481-10 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:137: oid 10 updating version 0 to 21 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:137: oid 10 version 21 is already newer than 21 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:137: oid 10 version 21 is already newer than 18 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:137: oid 10 version 21 is already newer than 21 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 10 v 21 (ObjNum 87 snap 9 seq_num 87) dirty exists 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:137: left oid 10 (ObjNum 87 snap 9 seq_num 87) 2026-02-25T08:42:08.323 INFO:tasks.rados.rados.0.trial039.stdout:140: finishing write tid 1 to trial03942481-31 2026-02-25T08:42:08.324 INFO:tasks.rados.rados.0.trial039.stdout:138: finishing write tid 2 to trial03942481-24 2026-02-25T08:42:08.335 INFO:tasks.rados.rados.0.trial039.stdout:138: finishing write tid 3 to trial03942481-24 2026-02-25T08:42:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:08.339 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:08.340 INFO:tasks.rados.rados.0.trial039.stdout:138: finishing write tid 4 to trial03942481-24 2026-02-25T08:42:08.345 INFO:tasks.rados.rados.0.trial039.stdout:138: finishing write tid 5 to trial03942481-24 2026-02-25T08:42:08.345 INFO:tasks.rados.rados.0.trial039.stdout:138: oid 24 updating version 0 to 21 2026-02-25T08:42:08.345 INFO:tasks.rados.rados.0.trial039.stdout:138: oid 24 version 21 is already newer than 21 2026-02-25T08:42:08.345 INFO:tasks.rados.rados.0.trial039.stdout:138: oid 24 version 21 is already newer than 21 2026-02-25T08:42:08.345 INFO:tasks.rados.rados.0.trial039.stdout:138: oid 24 version 21 is already newer than 18 2026-02-25T08:42:08.345 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 24 v 21 (ObjNum 88 snap 9 seq_num 88) dirty exists 2026-02-25T08:42:08.346 INFO:tasks.rados.rados.0.trial039.stdout:138: left oid 24 (ObjNum 88 snap 9 seq_num 88) 2026-02-25T08:42:08.346 INFO:tasks.rados.rados.0.trial039.stdout:141: finishing write tid 1 to trial03942481-28 2026-02-25T08:42:08.347 INFO:tasks.rados.rados.0.trial039.stdout:140: finishing write tid 2 to trial03942481-31 2026-02-25T08:42:08.352 INFO:tasks.rados.rados.0.trial039.stdout:140: finishing write tid 3 to trial03942481-31 2026-02-25T08:42:08.354 INFO:tasks.rados.rados.0.trial039.stdout:141: finishing write tid 2 to trial03942481-28 2026-02-25T08:42:08.356 INFO:tasks.rados.rados.0.trial039.stdout:140: finishing write tid 4 to trial03942481-31 2026-02-25T08:42:08.357 INFO:tasks.rados.rados.0.trial039.stdout:140: finishing write tid 5 to trial03942481-31 2026-02-25T08:42:08.357 INFO:tasks.rados.rados.0.trial039.stdout:140: oid 31 updating version 0 to 52 2026-02-25T08:42:08.357 INFO:tasks.rados.rados.0.trial039.stdout:140: oid 31 version 52 is already newer than 49 2026-02-25T08:42:08.357 INFO:tasks.rados.rados.0.trial039.stdout:140: oid 31 version 52 is already newer than 52 2026-02-25T08:42:08.357 INFO:tasks.rados.rados.0.trial039.stdout:140: oid 31 version 52 is already newer than 52 2026-02-25T08:42:08.357 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 52 (ObjNum 90 snap 9 seq_num 90) dirty exists 2026-02-25T08:42:08.358 INFO:tasks.rados.rados.0.trial039.stdout:140: left oid 31 (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:42:08.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~33s 2026-02-25T08:42:08.360 INFO:tasks.rados.rados.0.trial039.stdout:141: finishing write tid 3 to trial03942481-28 2026-02-25T08:42:08.364 INFO:tasks.rados.rados.0.trial039.stdout:141: finishing write tid 4 to trial03942481-28 2026-02-25T08:42:08.366 INFO:tasks.rados.rados.0.trial039.stdout:141: finishing write tid 5 to trial03942481-28 2026-02-25T08:42:08.366 INFO:tasks.rados.rados.0.trial039.stdout:141: oid 28 updating version 0 to 25 2026-02-25T08:42:08.366 INFO:tasks.rados.rados.0.trial039.stdout:141: oid 28 version 25 is already newer than 25 2026-02-25T08:42:08.366 INFO:tasks.rados.rados.0.trial039.stdout:141: oid 28 version 25 is already newer than 25 2026-02-25T08:42:08.366 INFO:tasks.rados.rados.0.trial039.stdout:141: oid 28 version 25 is already newer than 25 2026-02-25T08:42:08.367 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 28 v 25 (ObjNum 91 snap 9 seq_num 91) dirty exists 2026-02-25T08:42:08.367 INFO:tasks.rados.rados.0.trial039.stdout:141: left oid 28 (ObjNum 91 snap 9 seq_num 91) 2026-02-25T08:42:09.217 INFO:tasks.rados.rados.0.trial039.stdout:137: done (4 left) 2026-02-25T08:42:09.217 INFO:tasks.rados.rados.0.trial039.stdout:138: done (3 left) 2026-02-25T08:42:09.217 INFO:tasks.rados.rados.0.trial039.stdout:140: done (2 left) 2026-02-25T08:42:09.217 INFO:tasks.rados.rados.0.trial039.stdout:141: done (1 left) 2026-02-25T08:42:09.217 INFO:tasks.rados.rados.0.trial039.stdout:142: done (0 left) 2026-02-25T08:42:09.217 INFO:tasks.rados.rados.0.trial039.stdout:143: snap_create 2026-02-25T08:42:09.291 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:09.363 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:09.367 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:10.219 INFO:tasks.rados.rados.0.trial039.stdout:143: done (0 left) 2026-02-25T08:42:10.219 INFO:tasks.rados.rados.0.trial039.stdout:144: read oid 2 snap -1 2026-02-25T08:42:10.219 INFO:tasks.rados.rados.0.trial039.stdout:144: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:42:10.219 INFO:tasks.rados.rados.0.trial039.stdout:145: snap_remove snap 10 2026-02-25T08:42:10.340 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:10.420 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:10.416+0000 7fc186383640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:10.422 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:10.420+0000 7fc186383640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:10.450 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:10.466 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:11.225 INFO:tasks.rados.rados.0.trial039.stdout:145: done (1 left) 2026-02-25T08:42:11.225 INFO:tasks.rados.rados.0.trial039.stdout:146: write (excl) oid 40 current snap is 11 2026-02-25T08:42:11.225 INFO:tasks.rados.rados.0.trial039.stdout:146: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2026-02-25T08:42:11.230 INFO:tasks.rados.rados.0.trial039.stdout:146: writing trial03942481-40 from 746244 to 1465388 tid 1 2026-02-25T08:42:11.235 INFO:tasks.rados.rados.0.trial039.stdout:146: writing trial03942481-40 from 2100461 to 2603823 tid 2 2026-02-25T08:42:11.238 INFO:tasks.rados.rados.0.trial039.stdout:146: writing trial03942481-40 from 3126680 to 3546244 tid 3 2026-02-25T08:42:11.239 INFO:tasks.rados.rados.0.trial039.stdout:147: read oid 19 snap 9 2026-02-25T08:42:11.239 INFO:tasks.rados.rados.0.trial039.stdout:147: expect deleted 2026-02-25T08:42:11.239 INFO:tasks.rados.rados.0.trial039.stdout:148: write (excl) oid 34 current snap is 11 2026-02-25T08:42:11.239 INFO:tasks.rados.rados.0.trial039.stdout:148: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2026-02-25T08:42:11.244 INFO:tasks.rados.rados.0.trial039.stdout:148: writing trial03942481-34 from 763051 to 1557444 tid 1 2026-02-25T08:42:11.249 INFO:tasks.rados.rados.0.trial039.stdout:148: writing trial03942481-34 from 2158943 to 2805963 tid 2 2026-02-25T08:42:11.252 INFO:tasks.rados.rados.0.trial039.stdout:148: writing trial03942481-34 from 3437750 to 3563051 tid 3 2026-02-25T08:42:11.252 INFO:tasks.rados.rados.0.trial039.stdout:149: read oid 5 snap 9 2026-02-25T08:42:11.252 INFO:tasks.rados.rados.0.trial039.stdout:149: expect deleted 2026-02-25T08:42:11.252 INFO:tasks.rados.rados.0.trial039.stdout:150: read oid 38 snap -1 2026-02-25T08:42:11.253 INFO:tasks.rados.rados.0.trial039.stdout:150: expect (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:42:11.253 INFO:tasks.rados.rados.0.trial039.stdout:151: write oid 4 current snap is 11 2026-02-25T08:42:11.253 INFO:tasks.rados.rados.0.trial039.stdout:144: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:42:11.253 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:11.283 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:11.327 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1987890 2026-02-25T08:42:11.336 INFO:tasks.rados.rados.0.trial039.stdout:146: finishing write tid 1 to trial03942481-40 2026-02-25T08:42:11.336 INFO:tasks.rados.rados.0.trial039.stdout:150: expect (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:42:11.401 INFO:tasks.rados.rados.0.trial039.stdout:146: finishing write tid 2 to trial03942481-40 2026-02-25T08:42:11.401 INFO:tasks.rados.rados.0.trial039.stdout:146: finishing write tid 3 to trial03942481-40 2026-02-25T08:42:11.401 INFO:tasks.rados.rados.0.trial039.stdout:146: finishing write tid 4 to trial03942481-40 2026-02-25T08:42:11.401 INFO:tasks.rados.rados.0.trial039.stdout:146: finishing write tid 5 to trial03942481-40 2026-02-25T08:42:11.401 INFO:tasks.rados.rados.0.trial039.stdout:146: oid 40 updating version 0 to 40 2026-02-25T08:42:11.402 INFO:tasks.rados.rados.0.trial039.stdout:146: oid 40 version 40 is already newer than 40 2026-02-25T08:42:11.402 INFO:tasks.rados.rados.0.trial039.stdout:146: oid 40 version 40 is already newer than 37 2026-02-25T08:42:11.402 INFO:tasks.rados.rados.0.trial039.stdout:146: oid 40 version 40 is already newer than 40 2026-02-25T08:42:11.402 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 40 v 40 (ObjNum 92 snap 11 seq_num 92) dirty exists 2026-02-25T08:42:11.402 INFO:tasks.rados.rados.0.trial039.stdout:146: left oid 40 (ObjNum 92 snap 11 seq_num 92) 2026-02-25T08:42:11.402 INFO:tasks.rados.rados.0.trial039.stdout:148: finishing write tid 1 to trial03942481-34 2026-02-25T08:42:11.402 INFO:tasks.rados.rados.0.trial039.stdout:148: finishing write tid 2 to trial03942481-34 2026-02-25T08:42:11.402 INFO:tasks.rados.rados.0.trial039.stdout:151: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2026-02-25T08:42:11.406 INFO:tasks.rados.rados.0.trial039.stdout:151: writing trial03942481-4 from 779858 to 1249500 tid 1 2026-02-25T08:42:11.410 INFO:tasks.rados.rados.0.trial039.stdout:151: writing trial03942481-4 from 1817425 to 2324456 tid 2 2026-02-25T08:42:11.415 INFO:tasks.rados.rados.0.trial039.stdout:151: writing trial03942481-4 from 2781526 to 3368154 tid 3 2026-02-25T08:42:11.415 INFO:tasks.rados.rados.0.trial039.stdout:144: done (6 left) 2026-02-25T08:42:11.415 INFO:tasks.rados.rados.0.trial039.stdout:146: done (5 left) 2026-02-25T08:42:11.415 INFO:tasks.rados.rados.0.trial039.stdout:147: done (4 left) 2026-02-25T08:42:11.415 INFO:tasks.rados.rados.0.trial039.stdout:149: done (3 left) 2026-02-25T08:42:11.415 INFO:tasks.rados.rados.0.trial039.stdout:150: done (2 left) 2026-02-25T08:42:11.415 INFO:tasks.rados.rados.0.trial039.stdout:152: snap_remove snap 9 2026-02-25T08:42:12.230 INFO:tasks.rados.rados.0.trial039.stdout:152: done (2 left) 2026-02-25T08:42:12.230 INFO:tasks.rados.rados.0.trial039.stdout:153: delete oid 49 current snap is 11 2026-02-25T08:42:12.230 INFO:tasks.rados.rados.0.trial039.stdout:148: finishing write tid 3 to trial03942481-34 2026-02-25T08:42:12.230 INFO:tasks.rados.rados.0.trial039.stdout:148: finishing write tid 4 to trial03942481-34 2026-02-25T08:42:12.230 INFO:tasks.rados.rados.0.trial039.stdout:148: finishing write tid 5 to trial03942481-34 2026-02-25T08:42:12.230 INFO:tasks.rados.rados.0.trial039.stdout:148: oid 34 updating version 0 to 45 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:148: oid 34 version 45 is already newer than 45 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:148: oid 34 version 45 is already newer than 42 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:148: oid 34 version 45 is already newer than 45 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 45 (ObjNum 93 snap 11 seq_num 93) dirty exists 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:148: left oid 34 (ObjNum 93 snap 11 seq_num 93) 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:151: finishing write tid 1 to trial03942481-4 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:151: finishing write tid 2 to trial03942481-4 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:151: finishing write tid 3 to trial03942481-4 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:151: finishing write tid 4 to trial03942481-4 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:151: finishing write tid 5 to trial03942481-4 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:151: oid 4 updating version 0 to 26 2026-02-25T08:42:12.231 INFO:tasks.rados.rados.0.trial039.stdout:151: oid 4 version 26 is already newer than 26 2026-02-25T08:42:12.232 INFO:tasks.rados.rados.0.trial039.stdout:151: oid 4 version 26 is already newer than 26 2026-02-25T08:42:12.232 INFO:tasks.rados.rados.0.trial039.stdout:151: oid 4 version 26 is already newer than 26 2026-02-25T08:42:12.232 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 26 (ObjNum 94 snap 11 seq_num 94) dirty exists 2026-02-25T08:42:12.232 INFO:tasks.rados.rados.0.trial039.stdout:151: left oid 4 (ObjNum 94 snap 11 seq_num 94) 2026-02-25T08:42:12.255 INFO:tasks.rados.rados.0.trial039.stdout:148: done (2 left) 2026-02-25T08:42:12.256 INFO:tasks.rados.rados.0.trial039.stdout:151: done (1 left) 2026-02-25T08:42:12.256 INFO:tasks.rados.rados.0.trial039.stdout:153: done (0 left) 2026-02-25T08:42:12.256 INFO:tasks.rados.rados.0.trial039.stdout:154: delete oid 43 current snap is 11 2026-02-25T08:42:12.264 INFO:tasks.rados.rados.0.trial039.stdout:154: done (0 left) 2026-02-25T08:42:12.264 INFO:tasks.rados.rados.0.trial039.stdout:155: write (excl) oid 24 current snap is 11 2026-02-25T08:42:12.264 INFO:tasks.rados.rados.0.trial039.stdout:155: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2026-02-25T08:42:12.268 INFO:tasks.rados.rados.0.trial039.stdout:155: writing trial03942481-24 from 430279 to 1125668 tid 1 2026-02-25T08:42:12.274 INFO:tasks.rados.rados.0.trial039.stdout:155: writing trial03942481-24 from 1876518 to 2530876 tid 2 2026-02-25T08:42:12.279 INFO:tasks.rados.rados.0.trial039.stdout:155: writing trial03942481-24 from 3031089 to 3567886 tid 3 2026-02-25T08:42:12.279 INFO:tasks.rados.rados.0.trial039.stdout:156: read oid 31 snap -1 2026-02-25T08:42:12.279 INFO:tasks.rados.rados.0.trial039.stdout:156: expect (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:42:12.279 INFO:tasks.rados.rados.0.trial039.stdout:157: write (excl) oid 35 current snap is 11 2026-02-25T08:42:12.279 INFO:tasks.rados.rados.0.trial039.stdout:157: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2026-02-25T08:42:12.283 INFO:tasks.rados.rados.0.trial039.stdout:157: writing trial03942481-35 from 447086 to 1217724 tid 1 2026-02-25T08:42:12.289 INFO:tasks.rados.rados.0.trial039.stdout:157: writing trial03942481-35 from 1935000 to 2733016 tid 2 2026-02-25T08:42:12.292 INFO:tasks.rados.rados.0.trial039.stdout:157: writing trial03942481-35 from 3458512 to 3647086 tid 3 2026-02-25T08:42:12.293 INFO:tasks.rados.rados.0.trial039.stdout:158: write oid 38 current snap is 11 2026-02-25T08:42:12.293 INFO:tasks.rados.rados.0.trial039.stdout:158: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2026-02-25T08:42:12.296 INFO:tasks.rados.rados.0.trial039.stdout:158: writing trial03942481-38 from 463893 to 1026133 tid 1 2026-02-25T08:42:12.301 INFO:tasks.rados.rados.0.trial039.stdout:158: writing trial03942481-38 from 1593482 to 2251509 tid 2 2026-02-25T08:42:12.305 INFO:tasks.rados.rados.0.trial039.stdout:158: writing trial03942481-38 from 2685935 to 3245276 tid 3 2026-02-25T08:42:12.306 INFO:tasks.rados.rados.0.trial039.stdout:159: write oid 7 current snap is 11 2026-02-25T08:42:12.306 INFO:tasks.rados.rados.0.trial039.stdout:155: finishing write tid 1 to trial03942481-24 2026-02-25T08:42:12.306 INFO:tasks.rados.rados.0.trial039.stdout:155: finishing write tid 2 to trial03942481-24 2026-02-25T08:42:12.306 INFO:tasks.rados.rados.0.trial039.stdout:159: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2026-02-25T08:42:12.309 INFO:tasks.rados.rados.0.trial039.stdout:159: writing trial03942481-7 from 480700 to 1118189 tid 1 2026-02-25T08:42:12.313 INFO:tasks.rados.rados.0.trial039.stdout:159: writing trial03942481-7 from 1651964 to 2053649 tid 2 2026-02-25T08:42:12.319 INFO:tasks.rados.rados.0.trial039.stdout:159: writing trial03942481-7 from 2713358 to 3483971 tid 3 2026-02-25T08:42:12.319 INFO:tasks.rados.rados.0.trial039.stdout:160: write oid 20 current snap is 11 2026-02-25T08:42:12.319 INFO:tasks.rados.rados.0.trial039.stdout:156: expect (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:42:12.319 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 424270 2026-02-25T08:42:12.350 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1404709 2026-02-25T08:42:12.370 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 317265 2026-02-25T08:42:12.399 INFO:tasks.rados.rados.0.trial039.stdout:155: finishing write tid 3 to trial03942481-24 2026-02-25T08:42:12.399 INFO:tasks.rados.rados.0.trial039.stdout:157: finishing write tid 1 to trial03942481-35 2026-02-25T08:42:12.399 INFO:tasks.rados.rados.0.trial039.stdout:160: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2026-02-25T08:42:12.404 INFO:tasks.rados.rados.0.trial039.stdout:160: writing trial03942481-20 from 497507 to 1210245 tid 1 2026-02-25T08:42:12.408 INFO:tasks.rados.rados.0.trial039.stdout:160: writing trial03942481-20 from 1710446 to 2372142 tid 2 2026-02-25T08:42:12.414 INFO:tasks.rados.rados.0.trial039.stdout:160: writing trial03942481-20 from 3140781 to 3697507 tid 3 2026-02-25T08:42:12.414 INFO:tasks.rados.rados.0.trial039.stdout:156: done (5 left) 2026-02-25T08:42:12.414 INFO:tasks.rados.rados.0.trial039.stdout:161: write (excl) oid 1 current snap is 11 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:158: finishing write tid 1 to trial03942481-38 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:155: finishing write tid 4 to trial03942481-24 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:155: finishing write tid 5 to trial03942481-24 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:155: oid 24 updating version 0 to 22 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:155: oid 24 updating version 22 to 25 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:155: oid 24 version 25 is already newer than 25 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:155: oid 24 version 25 is already newer than 25 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 24 v 25 (ObjNum 97 snap 11 seq_num 97) dirty exists 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:155: left oid 24 (ObjNum 97 snap 11 seq_num 97) 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:158: finishing write tid 2 to trial03942481-38 2026-02-25T08:42:12.415 INFO:tasks.rados.rados.0.trial039.stdout:161: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2026-02-25T08:42:12.418 INFO:tasks.rados.rados.0.trial039.stdout:161: writing trial03942481-1 from 514314 to 1302301 tid 1 2026-02-25T08:42:12.422 INFO:tasks.rados.rados.0.trial039.stdout:161: writing trial03942481-1 from 1768928 to 2174282 tid 2 2026-02-25T08:42:12.427 INFO:tasks.rados.rados.0.trial039.stdout:161: writing trial03942481-1 from 2768204 to 3277714 tid 3 2026-02-25T08:42:12.427 INFO:tasks.rados.rados.0.trial039.stdout:155: done (5 left) 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:162: delete oid 32 current snap is 11 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:157: finishing write tid 2 to trial03942481-35 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:158: finishing write tid 3 to trial03942481-38 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:159: finishing write tid 1 to trial03942481-7 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:157: finishing write tid 3 to trial03942481-35 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:158: finishing write tid 4 to trial03942481-38 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:158: finishing write tid 5 to trial03942481-38 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:158: oid 38 updating version 0 to 30 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:158: oid 38 version 30 is already newer than 30 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:158: oid 38 version 30 is already newer than 28 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:158: oid 38 version 30 is already newer than 30 2026-02-25T08:42:12.428 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 38 v 30 (ObjNum 99 snap 11 seq_num 99) dirty exists 2026-02-25T08:42:12.429 INFO:tasks.rados.rados.0.trial039.stdout:158: left oid 38 (ObjNum 99 snap 11 seq_num 99) 2026-02-25T08:42:12.429 INFO:tasks.rados.rados.0.trial039.stdout:159: finishing write tid 2 to trial03942481-7 2026-02-25T08:42:12.429 INFO:tasks.rados.rados.0.trial039.stdout:157: finishing write tid 4 to trial03942481-35 2026-02-25T08:42:12.429 INFO:tasks.rados.rados.0.trial039.stdout:157: finishing write tid 5 to trial03942481-35 2026-02-25T08:42:12.429 INFO:tasks.rados.rados.0.trial039.stdout:157: oid 35 updating version 0 to 18 2026-02-25T08:42:12.429 INFO:tasks.rados.rados.0.trial039.stdout:157: oid 35 version 18 is already newer than 18 2026-02-25T08:42:12.429 INFO:tasks.rados.rados.0.trial039.stdout:157: oid 35 version 18 is already newer than 15 2026-02-25T08:42:12.429 INFO:tasks.rados.rados.0.trial039.stdout:157: oid 35 version 18 is already newer than 18 2026-02-25T08:42:12.429 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 35 v 18 (ObjNum 98 snap 11 seq_num 98) dirty exists 2026-02-25T08:42:12.429 INFO:tasks.rados.rados.0.trial039.stdout:157: left oid 35 (ObjNum 98 snap 11 seq_num 98) 2026-02-25T08:42:12.430 INFO:tasks.rados.rados.0.trial039.stdout:159: finishing write tid 3 to trial03942481-7 2026-02-25T08:42:12.430 INFO:tasks.rados.rados.0.trial039.stdout:159: finishing write tid 4 to trial03942481-7 2026-02-25T08:42:12.430 INFO:tasks.rados.rados.0.trial039.stdout:159: finishing write tid 5 to trial03942481-7 2026-02-25T08:42:12.430 INFO:tasks.rados.rados.0.trial039.stdout:159: oid 7 updating version 0 to 32 2026-02-25T08:42:12.430 INFO:tasks.rados.rados.0.trial039.stdout:159: oid 7 version 32 is already newer than 32 2026-02-25T08:42:12.430 INFO:tasks.rados.rados.0.trial039.stdout:159: oid 7 version 32 is already newer than 32 2026-02-25T08:42:12.430 INFO:tasks.rados.rados.0.trial039.stdout:159: oid 7 version 32 is already newer than 32 2026-02-25T08:42:12.430 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 7 v 32 (ObjNum 100 snap 11 seq_num 100) dirty exists 2026-02-25T08:42:12.430 INFO:tasks.rados.rados.0.trial039.stdout:159: left oid 7 (ObjNum 100 snap 11 seq_num 100) 2026-02-25T08:42:12.431 INFO:tasks.rados.rados.0.trial039.stdout:160: finishing write tid 1 to trial03942481-20 2026-02-25T08:42:12.447 INFO:tasks.rados.rados.0.trial039.stdout:157: done (5 left) 2026-02-25T08:42:12.447 INFO:tasks.rados.rados.0.trial039.stdout:158: done (4 left) 2026-02-25T08:42:12.447 INFO:tasks.rados.rados.0.trial039.stdout:159: done (3 left) 2026-02-25T08:42:12.447 INFO:tasks.rados.rados.0.trial039.stdout:162: done (2 left) 2026-02-25T08:42:12.447 INFO:tasks.rados.rados.0.trial039.stdout:163: read oid 22 snap -1 2026-02-25T08:42:12.448 INFO:tasks.rados.rados.0.trial039.stdout:163: expect (ObjNum 21 snap 0 seq_num 21) 2026-02-25T08:42:12.448 INFO:tasks.rados.rados.0.trial039.stdout:164: write oid 31 current snap is 11 2026-02-25T08:42:12.448 INFO:tasks.rados.rados.0.trial039.stdout:164: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2026-02-25T08:42:12.451 INFO:tasks.rados.rados.0.trial039.stdout:164: writing trial03942481-31 from 547928 to 1086413 tid 1 2026-02-25T08:42:12.455 INFO:tasks.rados.rados.0.trial039.stdout:164: writing trial03942481-31 from 1769539 to 2178562 tid 2 2026-02-25T08:42:12.460 INFO:tasks.rados.rados.0.trial039.stdout:164: writing trial03942481-31 from 2706697 to 3238751 tid 3 2026-02-25T08:42:12.460 INFO:tasks.rados.rados.0.trial039.stdout:165: delete oid 46 current snap is 11 2026-02-25T08:42:12.460 INFO:tasks.rados.rados.0.trial039.stdout:160: finishing write tid 2 to trial03942481-20 2026-02-25T08:42:12.460 INFO:tasks.rados.rados.0.trial039.stdout:160: finishing write tid 3 to trial03942481-20 2026-02-25T08:42:12.462 INFO:tasks.rados.rados.0.trial039.stdout:163: expect (ObjNum 21 snap 0 seq_num 21) 2026-02-25T08:42:12.462 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 219893 2026-02-25T08:42:12.467 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:12.501 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:12.522 INFO:tasks.rados.rados.0.trial039.stdout:160: finishing write tid 4 to trial03942481-20 2026-02-25T08:42:12.523 INFO:tasks.rados.rados.0.trial039.stdout:160: finishing write tid 5 to trial03942481-20 2026-02-25T08:42:12.523 INFO:tasks.rados.rados.0.trial039.stdout:160: oid 20 updating version 0 to 49 2026-02-25T08:42:12.523 INFO:tasks.rados.rados.0.trial039.stdout:160: oid 20 version 49 is already newer than 49 2026-02-25T08:42:12.523 INFO:tasks.rados.rados.0.trial039.stdout:160: oid 20 version 49 is already newer than 49 2026-02-25T08:42:12.523 INFO:tasks.rados.rados.0.trial039.stdout:160: oid 20 version 49 is already newer than 49 2026-02-25T08:42:12.523 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 20 v 49 (ObjNum 101 snap 11 seq_num 101) dirty exists 2026-02-25T08:42:12.523 INFO:tasks.rados.rados.0.trial039.stdout:160: left oid 20 (ObjNum 101 snap 11 seq_num 101) 2026-02-25T08:42:12.523 INFO:tasks.rados.rados.0.trial039.stdout:164: finishing write tid 1 to trial03942481-31 2026-02-25T08:42:12.523 INFO:tasks.rados.rados.0.trial039.stdout:161: finishing write tid 1 to trial03942481-1 2026-02-25T08:42:12.523 INFO:tasks.rados.rados.0.trial039.stdout:160: done (4 left) 2026-02-25T08:42:12.524 INFO:tasks.rados.rados.0.trial039.stdout:163: done (3 left) 2026-02-25T08:42:12.524 INFO:tasks.rados.rados.0.trial039.stdout:165: done (2 left) 2026-02-25T08:42:12.524 INFO:tasks.rados.rados.0.trial039.stdout:166: read oid 17 snap -1 2026-02-25T08:42:12.524 INFO:tasks.rados.rados.0.trial039.stdout:166: expect (ObjNum 68 snap 5 seq_num 68) 2026-02-25T08:42:12.524 INFO:tasks.rados.rados.0.trial039.stdout:167: write (excl) oid 49 current snap is 11 2026-02-25T08:42:12.524 INFO:tasks.rados.rados.0.trial039.stdout:164: finishing write tid 2 to trial03942481-31 2026-02-25T08:42:12.524 INFO:tasks.rados.rados.0.trial039.stdout:164: finishing write tid 3 to trial03942481-31 2026-02-25T08:42:12.524 INFO:tasks.rados.rados.0.trial039.stdout:164: finishing write tid 4 to trial03942481-31 2026-02-25T08:42:12.524 INFO:tasks.rados.rados.0.trial039.stdout:167: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2026-02-25T08:42:12.527 INFO:tasks.rados.rados.0.trial039.stdout:167: writing trial03942481-49 from 581542 to 1270525 tid 1 2026-02-25T08:42:12.531 INFO:tasks.rados.rados.0.trial039.stdout:167: writing trial03942481-49 from 1886503 to 2299195 tid 2 2026-02-25T08:42:12.536 INFO:tasks.rados.rados.0.trial039.stdout:167: writing trial03942481-49 from 2761543 to 3432494 tid 3 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:168: read oid 29 snap -1 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:168: expect (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:169: read oid 11 snap -1 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:169: expect (ObjNum 82 snap 9 seq_num 82) 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:170: read oid 26 snap -1 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:170: expect (ObjNum 75 snap 8 seq_num 75) 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:171: read oid 22 snap -1 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:171: expect (ObjNum 21 snap 0 seq_num 21) 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:172: read oid 25 snap -1 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:172: expect (ObjNum 78 snap 8 seq_num 78) 2026-02-25T08:42:12.537 INFO:tasks.rados.rados.0.trial039.stdout:173: write oid 2 current snap is 11 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:164: finishing write tid 5 to trial03942481-31 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:164: oid 31 updating version 0 to 56 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:164: oid 31 version 56 is already newer than 56 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:164: oid 31 version 56 is already newer than 56 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:164: oid 31 version 56 is already newer than 56 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 56 (ObjNum 104 snap 11 seq_num 104) dirty exists 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:164: left oid 31 (ObjNum 104 snap 11 seq_num 104) 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:161: finishing write tid 2 to trial03942481-1 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:161: finishing write tid 3 to trial03942481-1 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:161: finishing write tid 4 to trial03942481-1 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:161: finishing write tid 5 to trial03942481-1 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:161: oid 1 updating version 0 to 22 2026-02-25T08:42:12.538 INFO:tasks.rados.rados.0.trial039.stdout:161: oid 1 version 22 is already newer than 19 2026-02-25T08:42:12.539 INFO:tasks.rados.rados.0.trial039.stdout:161: oid 1 version 22 is already newer than 22 2026-02-25T08:42:12.539 INFO:tasks.rados.rados.0.trial039.stdout:161: oid 1 version 22 is already newer than 22 2026-02-25T08:42:12.539 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 1 v 22 (ObjNum 102 snap 11 seq_num 102) dirty exists 2026-02-25T08:42:12.539 INFO:tasks.rados.rados.0.trial039.stdout:161: left oid 1 (ObjNum 102 snap 11 seq_num 102) 2026-02-25T08:42:12.539 INFO:tasks.rados.rados.0.trial039.stdout:173: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2026-02-25T08:42:12.539 INFO:tasks.rados.rados.0.trial039.stdout:173: writing trial03942481-2 from 598349 to 1078934 tid 1 2026-02-25T08:42:12.544 INFO:tasks.rados.rados.0.trial039.stdout:173: writing trial03942481-2 from 1544985 to 2217688 tid 2 2026-02-25T08:42:12.544 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:12.541+0000 7f1beb05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:12.547 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:12.545+0000 7f1beb05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:12.548 INFO:tasks.rados.rados.0.trial039.stdout:173: writing trial03942481-2 from 2905319 to 3387542 tid 3 2026-02-25T08:42:12.548 INFO:tasks.rados.rados.0.trial039.stdout:161: done (9 left) 2026-02-25T08:42:12.548 INFO:tasks.rados.rados.0.trial039.stdout:164: done (8 left) 2026-02-25T08:42:12.549 INFO:tasks.rados.rados.0.trial039.stdout:174: snap_create 2026-02-25T08:42:12.550 INFO:tasks.rados.rados.0.trial039.stdout:171: expect (ObjNum 21 snap 0 seq_num 21) 2026-02-25T08:42:12.567 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1203882 2026-02-25T08:42:12.577 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 710499 2026-02-25T08:42:12.593 INFO:tasks.rados.rados.0.trial039.stdout:172: expect (ObjNum 78 snap 8 seq_num 78) 2026-02-25T08:42:12.604 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:12.620 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:12.620 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:12.632 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:12.702 INFO:tasks.rados.rados.0.trial039.stdout:169: expect (ObjNum 82 snap 9 seq_num 82) 2026-02-25T08:42:12.798 INFO:tasks.rados.rados.0.trial039.stdout:168: expect (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:42:12.798 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 444037 2026-02-25T08:42:12.865 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:12.899 INFO:tasks.rados.rados.0.trial039.stdout:167: finishing write tid 1 to trial03942481-49 2026-02-25T08:42:12.899 INFO:tasks.rados.rados.0.trial039.stdout:173: finishing write tid 1 to trial03942481-2 2026-02-25T08:42:12.900 INFO:tasks.rados.rados.0.trial039.stdout:166: expect (ObjNum 68 snap 5 seq_num 68) 2026-02-25T08:42:12.938 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:12.973 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:13.009 INFO:tasks.rados.rados.0.trial039.stdout:170: expect (ObjNum 75 snap 8 seq_num 75) 2026-02-25T08:42:13.047 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:13.101 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 3002634 2026-02-25T08:42:13.108 INFO:tasks.rados.rados.0.trial039.stdout:167: finishing write tid 2 to trial03942481-49 2026-02-25T08:42:13.108 INFO:tasks.rados.rados.0.trial039.stdout:167: finishing write tid 3 to trial03942481-49 2026-02-25T08:42:13.108 INFO:tasks.rados.rados.0.trial039.stdout:167: finishing write tid 4 to trial03942481-49 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:167: finishing write tid 5 to trial03942481-49 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:167: oid 49 updating version 0 to 29 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:167: oid 49 version 29 is already newer than 29 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:167: oid 49 version 29 is already newer than 29 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:167: oid 49 version 29 is already newer than 26 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 49 v 29 (ObjNum 106 snap 11 seq_num 106) dirty exists 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:167: left oid 49 (ObjNum 106 snap 11 seq_num 106) 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:173: finishing write tid 2 to trial03942481-2 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:173: finishing write tid 3 to trial03942481-2 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:173: finishing write tid 4 to trial03942481-2 2026-02-25T08:42:13.109 INFO:tasks.rados.rados.0.trial039.stdout:173: finishing write tid 5 to trial03942481-2 2026-02-25T08:42:13.110 INFO:tasks.rados.rados.0.trial039.stdout:173: oid 2 updating version 0 to 36 2026-02-25T08:42:13.110 INFO:tasks.rados.rados.0.trial039.stdout:173: oid 2 version 36 is already newer than 36 2026-02-25T08:42:13.110 INFO:tasks.rados.rados.0.trial039.stdout:173: oid 2 version 36 is already newer than 36 2026-02-25T08:42:13.110 INFO:tasks.rados.rados.0.trial039.stdout:173: oid 2 version 36 is already newer than 36 2026-02-25T08:42:13.110 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 36 (ObjNum 107 snap 11 seq_num 107) dirty exists 2026-02-25T08:42:13.110 INFO:tasks.rados.rados.0.trial039.stdout:173: left oid 2 (ObjNum 107 snap 11 seq_num 107) 2026-02-25T08:42:13.231 INFO:tasks.rados.rados.0.trial039.stdout:166: done (8 left) 2026-02-25T08:42:13.232 INFO:tasks.rados.rados.0.trial039.stdout:167: done (7 left) 2026-02-25T08:42:13.232 INFO:tasks.rados.rados.0.trial039.stdout:168: done (6 left) 2026-02-25T08:42:13.232 INFO:tasks.rados.rados.0.trial039.stdout:169: done (5 left) 2026-02-25T08:42:13.232 INFO:tasks.rados.rados.0.trial039.stdout:170: done (4 left) 2026-02-25T08:42:13.232 INFO:tasks.rados.rados.0.trial039.stdout:171: done (3 left) 2026-02-25T08:42:13.232 INFO:tasks.rados.rados.0.trial039.stdout:172: done (2 left) 2026-02-25T08:42:13.232 INFO:tasks.rados.rados.0.trial039.stdout:173: done (1 left) 2026-02-25T08:42:13.232 INFO:tasks.rados.rados.0.trial039.stdout:174: done (0 left) 2026-02-25T08:42:13.232 INFO:tasks.rados.rados.0.trial039.stdout:175: write (excl) oid 49 current snap is 12 2026-02-25T08:42:13.232 INFO:tasks.rados.rados.0.trial039.stdout:175: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2026-02-25T08:42:13.235 INFO:tasks.rados.rados.0.trial039.stdout:175: writing trial03942481-49 from 615156 to 1170990 tid 1 2026-02-25T08:42:13.239 INFO:tasks.rados.rados.0.trial039.stdout:175: writing trial03942481-49 from 1603467 to 2019828 tid 2 2026-02-25T08:42:13.245 INFO:tasks.rados.rados.0.trial039.stdout:175: writing trial03942481-49 from 2816389 to 3509884 tid 3 2026-02-25T08:42:13.246 INFO:tasks.rados.rados.0.trial039.stdout:176: snap_create 2026-02-25T08:42:13.268 INFO:tasks.rados.rados.0.trial039.stdout:175: finishing write tid 1 to trial03942481-49 2026-02-25T08:42:13.293 INFO:tasks.rados.rados.0.trial039.stdout:175: finishing write tid 2 to trial03942481-49 2026-02-25T08:42:13.313 INFO:tasks.rados.rados.0.trial039.stdout:175: finishing write tid 3 to trial03942481-49 2026-02-25T08:42:13.329 INFO:tasks.rados.rados.0.trial039.stdout:175: finishing write tid 4 to trial03942481-49 2026-02-25T08:42:13.331 INFO:tasks.rados.rados.0.trial039.stdout:175: finishing write tid 5 to trial03942481-49 2026-02-25T08:42:13.331 INFO:tasks.rados.rados.0.trial039.stdout:175: oid 49 updating version 0 to 34 2026-02-25T08:42:13.332 INFO:tasks.rados.rados.0.trial039.stdout:175: oid 49 version 34 is already newer than 34 2026-02-25T08:42:13.332 INFO:tasks.rados.rados.0.trial039.stdout:175: oid 49 version 34 is already newer than 34 2026-02-25T08:42:13.332 INFO:tasks.rados.rados.0.trial039.stdout:175: oid 49 version 34 is already newer than 31 2026-02-25T08:42:13.332 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 49 v 34 (ObjNum 108 snap 12 seq_num 108) dirty exists 2026-02-25T08:42:13.332 INFO:tasks.rados.rados.0.trial039.stdout:175: left oid 49 (ObjNum 108 snap 12 seq_num 108) 2026-02-25T08:42:13.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~39s 2026-02-25T08:42:14.235 INFO:tasks.rados.rados.0.trial039.stdout:175: done (1 left) 2026-02-25T08:42:14.235 INFO:tasks.rados.rados.0.trial039.stdout:176: done (0 left) 2026-02-25T08:42:14.235 INFO:tasks.rados.rados.0.trial039.stdout:177: read oid 49 snap -1 2026-02-25T08:42:14.235 INFO:tasks.rados.rados.0.trial039.stdout:177: expect (ObjNum 108 snap 12 seq_num 108) 2026-02-25T08:42:14.235 INFO:tasks.rados.rados.0.trial039.stdout:178: read oid 29 snap -1 2026-02-25T08:42:14.235 INFO:tasks.rados.rados.0.trial039.stdout:178: expect (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:42:14.235 INFO:tasks.rados.rados.0.trial039.stdout:179: snap_remove snap 12 2026-02-25T08:42:14.368 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:14.441 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:14.444 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:14.621 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:14.700 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:14.697+0000 7fa079411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:14.702 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:14.700+0000 7fa079411640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:14.729 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:14.746 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:15.240 INFO:tasks.rados.rados.0.trial039.stdout:179: done (2 left) 2026-02-25T08:42:15.240 INFO:tasks.rados.rados.0.trial039.stdout:180: write (excl) oid 7 current snap is 13 2026-02-25T08:42:15.240 INFO:tasks.rados.rados.0.trial039.stdout:180: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2026-02-25T08:42:15.245 INFO:tasks.rados.rados.0.trial039.stdout:180: writing trial03942481-7 from 631963 to 1263046 tid 1 2026-02-25T08:42:15.251 INFO:tasks.rados.rados.0.trial039.stdout:180: writing trial03942481-7 from 2061949 to 2621968 tid 2 2026-02-25T08:42:15.256 INFO:tasks.rados.rados.0.trial039.stdout:180: writing trial03942481-7 from 3243812 to 3748579 tid 3 2026-02-25T08:42:15.256 INFO:tasks.rados.rados.0.trial039.stdout:181: delete oid 2 current snap is 13 2026-02-25T08:42:15.256 INFO:tasks.rados.rados.0.trial039.stdout:178: expect (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:42:15.283 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 53747 2026-02-25T08:42:15.317 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:15.351 INFO:tasks.rados.rados.0.trial039.stdout:178: done (3 left) 2026-02-25T08:42:15.351 INFO:tasks.rados.rados.0.trial039.stdout:181: done (2 left) 2026-02-25T08:42:15.351 INFO:tasks.rados.rados.0.trial039.stdout:182: snap_remove snap 11 2026-02-25T08:42:16.242 INFO:tasks.rados.rados.0.trial039.stdout:182: done (2 left) 2026-02-25T08:42:16.242 INFO:tasks.rados.rados.0.trial039.stdout:183: delete oid 9 current snap is 13 2026-02-25T08:42:16.242 INFO:tasks.rados.rados.0.trial039.stdout:177: expect (ObjNum 108 snap 12 seq_num 108) 2026-02-25T08:42:16.242 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:16.295 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 910345 2026-02-25T08:42:16.376 INFO:tasks.rados.rados.0.trial039.stdout:177: done (2 left) 2026-02-25T08:42:16.377 INFO:tasks.rados.rados.0.trial039.stdout:183: done (1 left) 2026-02-25T08:42:16.377 INFO:tasks.rados.rados.0.trial039.stdout:184: write (excl) oid 3 current snap is 13 2026-02-25T08:42:16.377 INFO:tasks.rados.rados.0.trial039.stdout:180: finishing write tid 1 to trial03942481-7 2026-02-25T08:42:16.377 INFO:tasks.rados.rados.0.trial039.stdout:180: finishing write tid 2 to trial03942481-7 2026-02-25T08:42:16.377 INFO:tasks.rados.rados.0.trial039.stdout:180: finishing write tid 3 to trial03942481-7 2026-02-25T08:42:16.377 INFO:tasks.rados.rados.0.trial039.stdout:180: finishing write tid 4 to trial03942481-7 2026-02-25T08:42:16.377 INFO:tasks.rados.rados.0.trial039.stdout:180: finishing write tid 5 to trial03942481-7 2026-02-25T08:42:16.377 INFO:tasks.rados.rados.0.trial039.stdout:180: oid 7 updating version 0 to 38 2026-02-25T08:42:16.377 INFO:tasks.rados.rados.0.trial039.stdout:180: oid 7 updating version 38 to 41 2026-02-25T08:42:16.377 INFO:tasks.rados.rados.0.trial039.stdout:180: oid 7 version 41 is already newer than 41 2026-02-25T08:42:16.378 INFO:tasks.rados.rados.0.trial039.stdout:180: oid 7 version 41 is already newer than 41 2026-02-25T08:42:16.378 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 7 v 41 (ObjNum 109 snap 13 seq_num 109) dirty exists 2026-02-25T08:42:16.378 INFO:tasks.rados.rados.0.trial039.stdout:180: left oid 7 (ObjNum 109 snap 13 seq_num 109) 2026-02-25T08:42:16.378 INFO:tasks.rados.rados.0.trial039.stdout:184: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2026-02-25T08:42:16.380 INFO:tasks.rados.rados.0.trial039.stdout:184: writing trial03942481-3 from 682384 to 1139214 tid 1 2026-02-25T08:42:16.384 INFO:tasks.rados.rados.0.trial039.stdout:184: writing trial03942481-3 from 1721042 to 2144741 tid 2 2026-02-25T08:42:16.389 INFO:tasks.rados.rados.0.trial039.stdout:184: writing trial03942481-3 from 2809728 to 3264664 tid 3 2026-02-25T08:42:16.391 INFO:tasks.rados.rados.0.trial039.stdout:184: writing trial03942481-3 from 3731357 to 3882384 tid 4 2026-02-25T08:42:16.392 INFO:tasks.rados.rados.0.trial039.stdout:180: done (1 left) 2026-02-25T08:42:16.392 INFO:tasks.rados.rados.0.trial039.stdout:185: write oid 28 current snap is 13 2026-02-25T08:42:16.392 INFO:tasks.rados.rados.0.trial039.stdout:185: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2026-02-25T08:42:16.396 INFO:tasks.rados.rados.0.trial039.stdout:185: writing trial03942481-28 from 699191 to 1231270 tid 1 2026-02-25T08:42:16.400 INFO:tasks.rados.rados.0.trial039.stdout:185: writing trial03942481-28 from 1779524 to 2346881 tid 2 2026-02-25T08:42:16.405 INFO:tasks.rados.rados.0.trial039.stdout:185: writing trial03942481-28 from 2837151 to 3503359 tid 3 2026-02-25T08:42:16.405 INFO:tasks.rados.rados.0.trial039.stdout:186: snap_create 2026-02-25T08:42:16.405 INFO:tasks.rados.rados.0.trial039.stdout:184: finishing write tid 1 to trial03942481-3 2026-02-25T08:42:16.415 INFO:tasks.rados.rados.0.trial039.stdout:184: finishing write tid 2 to trial03942481-3 2026-02-25T08:42:16.421 INFO:tasks.rados.rados.0.trial039.stdout:185: finishing write tid 1 to trial03942481-28 2026-02-25T08:42:16.422 INFO:tasks.rados.rados.0.trial039.stdout:184: finishing write tid 3 to trial03942481-3 2026-02-25T08:42:16.428 INFO:tasks.rados.rados.0.trial039.stdout:185: finishing write tid 2 to trial03942481-28 2026-02-25T08:42:16.432 INFO:tasks.rados.rados.0.trial039.stdout:184: finishing write tid 4 to trial03942481-3 2026-02-25T08:42:16.433 INFO:tasks.rados.rados.0.trial039.stdout:184: finishing write tid 5 to trial03942481-3 2026-02-25T08:42:16.434 INFO:tasks.rados.rados.0.trial039.stdout:184: finishing write tid 6 to trial03942481-3 2026-02-25T08:42:16.434 INFO:tasks.rados.rados.0.trial039.stdout:184: oid 3 updating version 0 to 61 2026-02-25T08:42:16.435 INFO:tasks.rados.rados.0.trial039.stdout:184: oid 3 version 61 is already newer than 61 2026-02-25T08:42:16.435 INFO:tasks.rados.rados.0.trial039.stdout:184: oid 3 version 61 is already newer than 61 2026-02-25T08:42:16.435 INFO:tasks.rados.rados.0.trial039.stdout:184: oid 3 version 61 is already newer than 61 2026-02-25T08:42:16.435 INFO:tasks.rados.rados.0.trial039.stdout:184: oid 3 version 61 is already newer than 57 2026-02-25T08:42:16.435 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 3 v 61 (ObjNum 112 snap 13 seq_num 112) dirty exists 2026-02-25T08:42:16.435 INFO:tasks.rados.rados.0.trial039.stdout:184: left oid 3 (ObjNum 112 snap 13 seq_num 112) 2026-02-25T08:42:16.435 INFO:tasks.rados.rados.0.trial039.stdout:185: finishing write tid 3 to trial03942481-28 2026-02-25T08:42:16.442 INFO:tasks.rados.rados.0.trial039.stdout:185: finishing write tid 4 to trial03942481-28 2026-02-25T08:42:16.443 INFO:tasks.rados.rados.0.trial039.stdout:185: finishing write tid 5 to trial03942481-28 2026-02-25T08:42:16.444 INFO:tasks.rados.rados.0.trial039.stdout:185: oid 28 updating version 0 to 30 2026-02-25T08:42:16.444 INFO:tasks.rados.rados.0.trial039.stdout:185: oid 28 version 30 is already newer than 30 2026-02-25T08:42:16.444 INFO:tasks.rados.rados.0.trial039.stdout:185: oid 28 version 30 is already newer than 30 2026-02-25T08:42:16.444 INFO:tasks.rados.rados.0.trial039.stdout:185: oid 28 version 30 is already newer than 30 2026-02-25T08:42:16.444 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 28 v 30 (ObjNum 113 snap 13 seq_num 113) dirty exists 2026-02-25T08:42:16.444 INFO:tasks.rados.rados.0.trial039.stdout:185: left oid 28 (ObjNum 113 snap 13 seq_num 113) 2026-02-25T08:42:16.747 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:16.825 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:16.822+0000 7f2f03bfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:16.827 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:16.825+0000 7f2f03bfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:16.853 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:16.871 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:17.249 INFO:tasks.rados.rados.0.trial039.stdout:184: done (2 left) 2026-02-25T08:42:17.249 INFO:tasks.rados.rados.0.trial039.stdout:185: done (1 left) 2026-02-25T08:42:17.249 INFO:tasks.rados.rados.0.trial039.stdout:186: done (0 left) 2026-02-25T08:42:17.249 INFO:tasks.rados.rados.0.trial039.stdout:187: write (excl) oid 45 current snap is 14 2026-02-25T08:42:17.249 INFO:tasks.rados.rados.0.trial039.stdout:187: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2026-02-25T08:42:17.254 INFO:tasks.rados.rados.0.trial039.stdout:187: writing trial03942481-45 from 715998 to 1323326 tid 1 2026-02-25T08:42:17.258 INFO:tasks.rados.rados.0.trial039.stdout:187: writing trial03942481-45 from 1838006 to 2265374 tid 2 2026-02-25T08:42:17.262 INFO:tasks.rados.rados.0.trial039.stdout:187: writing trial03942481-45 from 2864574 to 3342054 tid 3 2026-02-25T08:42:17.265 INFO:tasks.rados.rados.0.trial039.stdout:187: writing trial03942481-45 from 3863835 to 3915998 tid 4 2026-02-25T08:42:17.265 INFO:tasks.rados.rados.0.trial039.stdout:188: snap_remove snap 13 2026-02-25T08:42:18.250 INFO:tasks.rados.rados.0.trial039.stdout:188: done (1 left) 2026-02-25T08:42:18.250 INFO:tasks.rados.rados.0.trial039.stdout:189: write (excl) oid 9 current snap is 14 2026-02-25T08:42:18.250 INFO:tasks.rados.rados.0.trial039.stdout:187: finishing write tid 1 to trial03942481-45 2026-02-25T08:42:18.250 INFO:tasks.rados.rados.0.trial039.stdout:189: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2026-02-25T08:42:18.255 INFO:tasks.rados.rados.0.trial039.stdout:189: writing trial03942481-9 from 732805 to 1531735 tid 1 2026-02-25T08:42:18.261 INFO:tasks.rados.rados.0.trial039.stdout:189: writing trial03942481-9 from 2296488 to 2867514 tid 2 2026-02-25T08:42:18.265 INFO:tasks.rados.rados.0.trial039.stdout:189: writing trial03942481-9 from 3291997 to 3697102 tid 3 2026-02-25T08:42:18.265 INFO:tasks.rados.rados.0.trial039.stdout:190: write (excl) oid 20 current snap is 14 2026-02-25T08:42:18.265 INFO:tasks.rados.rados.0.trial039.stdout:187: finishing write tid 2 to trial03942481-45 2026-02-25T08:42:18.265 INFO:tasks.rados.rados.0.trial039.stdout:190: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2026-02-25T08:42:18.269 INFO:tasks.rados.rados.0.trial039.stdout:190: writing trial03942481-20 from 749612 to 1223791 tid 1 2026-02-25T08:42:18.273 INFO:tasks.rados.rados.0.trial039.stdout:190: writing trial03942481-20 from 1954970 to 2386007 tid 2 2026-02-25T08:42:18.278 INFO:tasks.rados.rados.0.trial039.stdout:190: writing trial03942481-20 from 2919420 to 3535797 tid 3 2026-02-25T08:42:18.278 INFO:tasks.rados.rados.0.trial039.stdout:191: snap_create 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:187: finishing write tid 3 to trial03942481-45 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:187: finishing write tid 4 to trial03942481-45 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:187: finishing write tid 5 to trial03942481-45 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:187: finishing write tid 6 to trial03942481-45 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:187: oid 45 updating version 0 to 55 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:187: oid 45 version 55 is already newer than 51 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:187: oid 45 version 55 is already newer than 55 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:187: oid 45 version 55 is already newer than 55 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:187: oid 45 version 55 is already newer than 55 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 45 v 55 (ObjNum 114 snap 14 seq_num 114) dirty exists 2026-02-25T08:42:18.279 INFO:tasks.rados.rados.0.trial039.stdout:187: left oid 45 (ObjNum 114 snap 14 seq_num 114) 2026-02-25T08:42:18.283 INFO:tasks.rados.rados.0.trial039.stdout:190: finishing write tid 1 to trial03942481-20 2026-02-25T08:42:18.285 INFO:tasks.rados.rados.0.trial039.stdout:189: finishing write tid 1 to trial03942481-9 2026-02-25T08:42:18.297 INFO:tasks.rados.rados.0.trial039.stdout:189: finishing write tid 2 to trial03942481-9 2026-02-25T08:42:18.297 INFO:tasks.rados.rados.0.trial039.stdout:190: finishing write tid 2 to trial03942481-20 2026-02-25T08:42:18.303 INFO:tasks.rados.rados.0.trial039.stdout:190: finishing write tid 3 to trial03942481-20 2026-02-25T08:42:18.303 INFO:tasks.rados.rados.0.trial039.stdout:189: finishing write tid 3 to trial03942481-9 2026-02-25T08:42:18.307 INFO:tasks.rados.rados.0.trial039.stdout:190: finishing write tid 4 to trial03942481-20 2026-02-25T08:42:18.308 INFO:tasks.rados.rados.0.trial039.stdout:189: finishing write tid 4 to trial03942481-9 2026-02-25T08:42:18.310 INFO:tasks.rados.rados.0.trial039.stdout:190: finishing write tid 5 to trial03942481-20 2026-02-25T08:42:18.310 INFO:tasks.rados.rados.0.trial039.stdout:190: oid 20 updating version 0 to 56 2026-02-25T08:42:18.310 INFO:tasks.rados.rados.0.trial039.stdout:190: oid 20 updating version 56 to 59 2026-02-25T08:42:18.310 INFO:tasks.rados.rados.0.trial039.stdout:190: oid 20 version 59 is already newer than 59 2026-02-25T08:42:18.310 INFO:tasks.rados.rados.0.trial039.stdout:190: oid 20 version 59 is already newer than 59 2026-02-25T08:42:18.310 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 20 v 59 (ObjNum 116 snap 14 seq_num 116) dirty exists 2026-02-25T08:42:18.310 INFO:tasks.rados.rados.0.trial039.stdout:190: left oid 20 (ObjNum 116 snap 14 seq_num 116) 2026-02-25T08:42:18.310 INFO:tasks.rados.rados.0.trial039.stdout:189: finishing write tid 5 to trial03942481-9 2026-02-25T08:42:18.310 INFO:tasks.rados.rados.0.trial039.stdout:189: oid 9 updating version 0 to 9 2026-02-25T08:42:18.310 INFO:tasks.rados.rados.0.trial039.stdout:189: oid 9 version 9 is already newer than 9 2026-02-25T08:42:18.311 INFO:tasks.rados.rados.0.trial039.stdout:189: oid 9 version 9 is already newer than 6 2026-02-25T08:42:18.311 INFO:tasks.rados.rados.0.trial039.stdout:189: oid 9 version 9 is already newer than 9 2026-02-25T08:42:18.311 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 9 v 9 (ObjNum 115 snap 14 seq_num 115) dirty exists 2026-02-25T08:42:18.311 INFO:tasks.rados.rados.0.trial039.stdout:189: left oid 9 (ObjNum 115 snap 14 seq_num 115) 2026-02-25T08:42:18.872 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:18.951 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:18.948+0000 7f0cf310b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:18.953 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:18.951+0000 7f0cf310b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:18.980 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:18.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:19.255 INFO:tasks.rados.rados.0.trial039.stdout:187: done (3 left) 2026-02-25T08:42:19.256 INFO:tasks.rados.rados.0.trial039.stdout:189: done (2 left) 2026-02-25T08:42:19.256 INFO:tasks.rados.rados.0.trial039.stdout:190: done (1 left) 2026-02-25T08:42:19.256 INFO:tasks.rados.rados.0.trial039.stdout:191: done (0 left) 2026-02-25T08:42:19.256 INFO:tasks.rados.rados.0.trial039.stdout:192: snap_create 2026-02-25T08:42:19.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~44s 2026-02-25T08:42:19.445 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:19.517 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:19.520 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial039.stdout:192: done (0 left) 2026-02-25T08:42:20.258 INFO:tasks.rados.rados.0.trial039.stdout:193: snap_remove snap 15 2026-02-25T08:42:20.997 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:21.074 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:21.071+0000 7f4ff78b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:21.076 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:21.075+0000 7f4ff78b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:21.109 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:21.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:21.262 INFO:tasks.rados.rados.0.trial039.stdout:193: done (0 left) 2026-02-25T08:42:21.263 INFO:tasks.rados.rados.0.trial039.stdout:194: write (excl) oid 9 current snap is 16 2026-02-25T08:42:21.263 INFO:tasks.rados.rados.0.trial039.stdout:194: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2026-02-25T08:42:21.268 INFO:tasks.rados.rados.0.trial039.stdout:194: writing trial03942481-9 from 766419 to 1315847 tid 1 2026-02-25T08:42:21.273 INFO:tasks.rados.rados.0.trial039.stdout:194: writing trial03942481-9 from 2013452 to 2588147 tid 2 2026-02-25T08:42:21.278 INFO:tasks.rados.rados.0.trial039.stdout:194: writing trial03942481-9 from 3346843 to 3774492 tid 3 2026-02-25T08:42:21.278 INFO:tasks.rados.rados.0.trial039.stdout:195: snap_create 2026-02-25T08:42:21.304 INFO:tasks.rados.rados.0.trial039.stdout:194: finishing write tid 1 to trial03942481-9 2026-02-25T08:42:21.320 INFO:tasks.rados.rados.0.trial039.stdout:194: finishing write tid 2 to trial03942481-9 2026-02-25T08:42:21.326 INFO:tasks.rados.rados.0.trial039.stdout:194: finishing write tid 3 to trial03942481-9 2026-02-25T08:42:21.331 INFO:tasks.rados.rados.0.trial039.stdout:194: finishing write tid 4 to trial03942481-9 2026-02-25T08:42:21.332 INFO:tasks.rados.rados.0.trial039.stdout:194: finishing write tid 5 to trial03942481-9 2026-02-25T08:42:21.332 INFO:tasks.rados.rados.0.trial039.stdout:194: oid 9 updating version 0 to 11 2026-02-25T08:42:21.332 INFO:tasks.rados.rados.0.trial039.stdout:194: oid 9 updating version 11 to 14 2026-02-25T08:42:21.333 INFO:tasks.rados.rados.0.trial039.stdout:194: oid 9 version 14 is already newer than 14 2026-02-25T08:42:21.333 INFO:tasks.rados.rados.0.trial039.stdout:194: oid 9 version 14 is already newer than 14 2026-02-25T08:42:21.333 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 9 v 14 (ObjNum 117 snap 16 seq_num 117) dirty exists 2026-02-25T08:42:21.333 INFO:tasks.rados.rados.0.trial039.stdout:194: left oid 9 (ObjNum 117 snap 16 seq_num 117) 2026-02-25T08:42:22.265 INFO:tasks.rados.rados.0.trial039.stdout:194: done (1 left) 2026-02-25T08:42:22.266 INFO:tasks.rados.rados.0.trial039.stdout:195: done (0 left) 2026-02-25T08:42:22.266 INFO:tasks.rados.rados.0.trial039.stdout:196: read oid 33 snap -1 2026-02-25T08:42:22.266 INFO:tasks.rados.rados.0.trial039.stdout:196: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-25T08:42:22.266 INFO:tasks.rados.rados.0.trial039.stdout:197: write (excl) oid 31 current snap is 17 2026-02-25T08:42:22.266 INFO:tasks.rados.rados.0.trial039.stdout:197: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2026-02-25T08:42:22.270 INFO:tasks.rados.rados.0.trial039.stdout:197: writing trial03942481-31 from 783226 to 1407903 tid 1 2026-02-25T08:42:22.275 INFO:tasks.rados.rados.0.trial039.stdout:197: writing trial03942481-31 from 2071934 to 2506640 tid 2 2026-02-25T08:42:22.280 INFO:tasks.rados.rados.0.trial039.stdout:197: writing trial03942481-31 from 2974266 to 3613187 tid 3 2026-02-25T08:42:22.280 INFO:tasks.rados.rados.0.trial039.stdout:198: delete oid 28 current snap is 17 2026-02-25T08:42:22.299 INFO:tasks.rados.rados.0.trial039.stdout:196: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-25T08:42:22.361 INFO:tasks.rados.rados.0.trial039.stdout:197: finishing write tid 1 to trial03942481-31 2026-02-25T08:42:22.362 INFO:tasks.rados.rados.0.trial039.stdout:197: finishing write tid 2 to trial03942481-31 2026-02-25T08:42:22.362 INFO:tasks.rados.rados.0.trial039.stdout:196: done (2 left) 2026-02-25T08:42:22.362 INFO:tasks.rados.rados.0.trial039.stdout:198: done (1 left) 2026-02-25T08:42:22.362 INFO:tasks.rados.rados.0.trial039.stdout:199: delete oid 42 current snap is 17 2026-02-25T08:42:22.362 INFO:tasks.rados.rados.0.trial039.stdout:197: finishing write tid 3 to trial03942481-31 2026-02-25T08:42:22.362 INFO:tasks.rados.rados.0.trial039.stdout:197: finishing write tid 4 to trial03942481-31 2026-02-25T08:42:22.362 INFO:tasks.rados.rados.0.trial039.stdout:197: finishing write tid 5 to trial03942481-31 2026-02-25T08:42:22.362 INFO:tasks.rados.rados.0.trial039.stdout:197: oid 31 updating version 0 to 66 2026-02-25T08:42:22.362 INFO:tasks.rados.rados.0.trial039.stdout:197: oid 31 version 66 is already newer than 63 2026-02-25T08:42:22.362 INFO:tasks.rados.rados.0.trial039.stdout:197: oid 31 version 66 is already newer than 66 2026-02-25T08:42:22.363 INFO:tasks.rados.rados.0.trial039.stdout:197: oid 31 version 66 is already newer than 66 2026-02-25T08:42:22.363 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 66 (ObjNum 118 snap 17 seq_num 118) dirty exists 2026-02-25T08:42:22.363 INFO:tasks.rados.rados.0.trial039.stdout:197: left oid 31 (ObjNum 118 snap 17 seq_num 118) 2026-02-25T08:42:22.384 INFO:tasks.rados.rados.0.trial039.stdout:197: done (1 left) 2026-02-25T08:42:22.385 INFO:tasks.rados.rados.0.trial039.stdout:199: done (0 left) 2026-02-25T08:42:22.385 INFO:tasks.rados.rados.0.trial039.stdout:200: read oid 3 snap -1 2026-02-25T08:42:22.385 INFO:tasks.rados.rados.0.trial039.stdout:200: expect (ObjNum 112 snap 13 seq_num 112) 2026-02-25T08:42:22.385 INFO:tasks.rados.rados.0.trial039.stdout:201: write (excl) oid 37 current snap is 17 2026-02-25T08:42:22.385 INFO:tasks.rados.rados.0.trial039.stdout:201: seq_num 121 ranges {433647=450424,1331027=582033} 2026-02-25T08:42:22.387 INFO:tasks.rados.rados.0.trial039.stdout:201: writing trial03942481-37 from 433647 to 884071 tid 1 2026-02-25T08:42:22.392 INFO:tasks.rados.rados.0.trial039.stdout:201: writing trial03942481-37 from 1331027 to 1913060 tid 2 2026-02-25T08:42:22.392 INFO:tasks.rados.rados.0.trial039.stdout:202: write (excl) oid 11 current snap is 17 2026-02-25T08:42:22.392 INFO:tasks.rados.rados.0.trial039.stdout:202: seq_num 122 ranges {450454=642026,1505862=544592} 2026-02-25T08:42:22.396 INFO:tasks.rados.rados.0.trial039.stdout:202: writing trial03942481-11 from 450454 to 1092480 tid 1 2026-02-25T08:42:22.400 INFO:tasks.rados.rados.0.trial039.stdout:202: writing trial03942481-11 from 1505862 to 2050454 tid 2 2026-02-25T08:42:22.400 INFO:tasks.rados.rados.0.trial039.stdout:203: snap_remove snap 14 2026-02-25T08:42:23.127 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:23.205 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:23.202+0000 7f922045a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:23.208 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:23.206+0000 7f922045a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:23.233 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:23.251 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:23.268 INFO:tasks.rados.rados.0.trial039.stdout:203: done (3 left) 2026-02-25T08:42:23.268 INFO:tasks.rados.rados.0.trial039.stdout:204: write (excl) oid 19 current snap is 17 2026-02-25T08:42:23.268 INFO:tasks.rados.rados.0.trial039.stdout:204: seq_num 123 ranges {467261=717275,1847991=219270} 2026-02-25T08:42:23.272 INFO:tasks.rados.rados.0.trial039.stdout:204: writing trial03942481-19 from 467261 to 1184536 tid 1 2026-02-25T08:42:23.276 INFO:tasks.rados.rados.0.trial039.stdout:204: writing trial03942481-19 from 1847991 to 2067261 tid 2 2026-02-25T08:42:23.276 INFO:tasks.rados.rados.0.trial039.stdout:205: write oid 23 current snap is 17 2026-02-25T08:42:23.276 INFO:tasks.rados.rados.0.trial039.stdout:200: expect (ObjNum 112 snap 13 seq_num 112) 2026-02-25T08:42:23.283 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2021752 2026-02-25T08:42:23.333 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:23.367 INFO:tasks.rados.rados.0.trial039.stdout:201: finishing write tid 1 to trial03942481-37 2026-02-25T08:42:23.367 INFO:tasks.rados.rados.0.trial039.stdout:205: seq_num 124 ranges {484068=792524,1906473=177595} 2026-02-25T08:42:23.372 INFO:tasks.rados.rados.0.trial039.stdout:205: writing trial03942481-23 from 484068 to 1276592 tid 1 2026-02-25T08:42:23.376 INFO:tasks.rados.rados.0.trial039.stdout:205: writing trial03942481-23 from 1906473 to 2084068 tid 2 2026-02-25T08:42:23.376 INFO:tasks.rados.rados.0.trial039.stdout:200: done (4 left) 2026-02-25T08:42:23.376 INFO:tasks.rados.rados.0.trial039.stdout:206: snap_create 2026-02-25T08:42:23.376 INFO:tasks.rados.rados.0.trial039.stdout:202: finishing write tid 1 to trial03942481-11 2026-02-25T08:42:23.376 INFO:tasks.rados.rados.0.trial039.stdout:201: finishing write tid 2 to trial03942481-37 2026-02-25T08:42:23.376 INFO:tasks.rados.rados.0.trial039.stdout:202: finishing write tid 2 to trial03942481-11 2026-02-25T08:42:23.376 INFO:tasks.rados.rados.0.trial039.stdout:201: finishing write tid 3 to trial03942481-37 2026-02-25T08:42:23.376 INFO:tasks.rados.rados.0.trial039.stdout:201: finishing write tid 4 to trial03942481-37 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:201: oid 37 updating version 0 to 45 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:201: oid 37 updating version 45 to 47 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:201: oid 37 version 47 is already newer than 47 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 37 v 47 (ObjNum 121 snap 17 seq_num 121) dirty exists 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:201: left oid 37 (ObjNum 121 snap 17 seq_num 121) 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:202: finishing write tid 3 to trial03942481-11 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:202: finishing write tid 4 to trial03942481-11 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:202: oid 11 updating version 0 to 12 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:202: oid 11 version 12 is already newer than 10 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:202: oid 11 version 12 is already newer than 12 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 11 v 12 (ObjNum 122 snap 17 seq_num 122) dirty exists 2026-02-25T08:42:23.377 INFO:tasks.rados.rados.0.trial039.stdout:202: left oid 11 (ObjNum 122 snap 17 seq_num 122) 2026-02-25T08:42:23.378 INFO:tasks.rados.rados.0.trial039.stdout:204: finishing write tid 1 to trial03942481-19 2026-02-25T08:42:23.378 INFO:tasks.rados.rados.0.trial039.stdout:204: finishing write tid 2 to trial03942481-19 2026-02-25T08:42:23.378 INFO:tasks.rados.rados.0.trial039.stdout:204: finishing write tid 3 to trial03942481-19 2026-02-25T08:42:23.378 INFO:tasks.rados.rados.0.trial039.stdout:204: finishing write tid 4 to trial03942481-19 2026-02-25T08:42:23.378 INFO:tasks.rados.rados.0.trial039.stdout:204: oid 19 updating version 0 to 69 2026-02-25T08:42:23.378 INFO:tasks.rados.rados.0.trial039.stdout:204: oid 19 updating version 69 to 75 2026-02-25T08:42:23.378 INFO:tasks.rados.rados.0.trial039.stdout:204: oid 19 version 75 is already newer than 75 2026-02-25T08:42:23.378 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 19 v 75 (ObjNum 123 snap 17 seq_num 123) dirty exists 2026-02-25T08:42:23.378 INFO:tasks.rados.rados.0.trial039.stdout:204: left oid 19 (ObjNum 123 snap 17 seq_num 123) 2026-02-25T08:42:23.400 INFO:tasks.rados.rados.0.trial039.stdout:205: finishing write tid 1 to trial03942481-23 2026-02-25T08:42:23.408 INFO:tasks.rados.rados.0.trial039.stdout:205: finishing write tid 2 to trial03942481-23 2026-02-25T08:42:23.415 INFO:tasks.rados.rados.0.trial039.stdout:205: finishing write tid 3 to trial03942481-23 2026-02-25T08:42:23.416 INFO:tasks.rados.rados.0.trial039.stdout:205: finishing write tid 4 to trial03942481-23 2026-02-25T08:42:23.417 INFO:tasks.rados.rados.0.trial039.stdout:205: oid 23 updating version 0 to 22 2026-02-25T08:42:23.417 INFO:tasks.rados.rados.0.trial039.stdout:205: oid 23 version 22 is already newer than 22 2026-02-25T08:42:23.417 INFO:tasks.rados.rados.0.trial039.stdout:205: oid 23 version 22 is already newer than 22 2026-02-25T08:42:23.417 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 23 v 22 (ObjNum 124 snap 17 seq_num 124) dirty exists 2026-02-25T08:42:23.417 INFO:tasks.rados.rados.0.trial039.stdout:205: left oid 23 (ObjNum 124 snap 17 seq_num 124) 2026-02-25T08:42:24.273 INFO:tasks.rados.rados.0.trial039.stdout:201: done (4 left) 2026-02-25T08:42:24.273 INFO:tasks.rados.rados.0.trial039.stdout:202: done (3 left) 2026-02-25T08:42:24.273 INFO:tasks.rados.rados.0.trial039.stdout:204: done (2 left) 2026-02-25T08:42:24.273 INFO:tasks.rados.rados.0.trial039.stdout:205: done (1 left) 2026-02-25T08:42:24.273 INFO:tasks.rados.rados.0.trial039.stdout:206: done (0 left) 2026-02-25T08:42:24.273 INFO:tasks.rados.rados.0.trial039.stdout:207: read oid 13 snap -1 2026-02-25T08:42:24.274 INFO:tasks.rados.rados.0.trial039.stdout:207: expect deleted 2026-02-25T08:42:24.274 INFO:tasks.rados.rados.0.trial039.stdout:208: delete oid 37 current snap is 18 2026-02-25T08:42:24.310 INFO:tasks.rados.rados.0.trial039.stdout:207: done (1 left) 2026-02-25T08:42:24.310 INFO:tasks.rados.rados.0.trial039.stdout:208: done (0 left) 2026-02-25T08:42:24.310 INFO:tasks.rados.rados.0.trial039.stdout:209: read oid 31 snap 16 2026-02-25T08:42:24.310 INFO:tasks.rados.rados.0.trial039.stdout:209: expect (ObjNum 104 snap 11 seq_num 104) 2026-02-25T08:42:24.310 INFO:tasks.rados.rados.0.trial039.stdout:210: read oid 10 snap -1 2026-02-25T08:42:24.311 INFO:tasks.rados.rados.0.trial039.stdout:210: expect (ObjNum 87 snap 9 seq_num 87) 2026-02-25T08:42:24.311 INFO:tasks.rados.rados.0.trial039.stdout:211: read oid 21 snap -1 2026-02-25T08:42:24.311 INFO:tasks.rados.rados.0.trial039.stdout:211: expect (ObjNum 20 snap 0 seq_num 20) 2026-02-25T08:42:24.311 INFO:tasks.rados.rados.0.trial039.stdout:212: write oid 25 current snap is 18 2026-02-25T08:42:24.311 INFO:tasks.rados.rados.0.trial039.stdout:212: seq_num 126 ranges {517682=543022,1623437=494245} 2026-02-25T08:42:24.313 INFO:tasks.rados.rados.0.trial039.stdout:212: writing trial03942481-25 from 517682 to 1060704 tid 1 2026-02-25T08:42:24.318 INFO:tasks.rados.rados.0.trial039.stdout:212: writing trial03942481-25 from 1623437 to 2117682 tid 2 2026-02-25T08:42:24.318 INFO:tasks.rados.rados.0.trial039.stdout:213: snap_create 2026-02-25T08:42:24.322 INFO:tasks.rados.rados.0.trial039.stdout:210: expect (ObjNum 87 snap 9 seq_num 87) 2026-02-25T08:42:24.334 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1891581 2026-02-25T08:42:24.419 INFO:tasks.rados.rados.0.trial039.stdout:209: expect (ObjNum 104 snap 11 seq_num 104) 2026-02-25T08:42:24.419 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:24.463 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:24.521 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:24.537 INFO:tasks.rados.rados.0.trial039.stdout:211: expect (ObjNum 20 snap 0 seq_num 20) 2026-02-25T08:42:24.537 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:24.557 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:24.578 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:24.592 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:24.595 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:24.598 INFO:tasks.rados.rados.0.trial039.stdout:212: finishing write tid 1 to trial03942481-25 2026-02-25T08:42:24.598 INFO:tasks.rados.rados.0.trial039.stdout:212: finishing write tid 2 to trial03942481-25 2026-02-25T08:42:24.598 INFO:tasks.rados.rados.0.trial039.stdout:212: finishing write tid 3 to trial03942481-25 2026-02-25T08:42:24.598 INFO:tasks.rados.rados.0.trial039.stdout:212: finishing write tid 4 to trial03942481-25 2026-02-25T08:42:24.598 INFO:tasks.rados.rados.0.trial039.stdout:212: oid 25 updating version 0 to 11 2026-02-25T08:42:24.598 INFO:tasks.rados.rados.0.trial039.stdout:212: oid 25 version 11 is already newer than 11 2026-02-25T08:42:24.598 INFO:tasks.rados.rados.0.trial039.stdout:212: oid 25 version 11 is already newer than 11 2026-02-25T08:42:24.598 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 25 v 11 (ObjNum 126 snap 18 seq_num 126) dirty exists 2026-02-25T08:42:24.599 INFO:tasks.rados.rados.0.trial039.stdout:212: left oid 25 (ObjNum 126 snap 18 seq_num 126) 2026-02-25T08:42:24.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2026-02-25T08:42:25.252 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:25.273 INFO:tasks.rados.rados.0.trial039.stdout:209: done (4 left) 2026-02-25T08:42:25.273 INFO:tasks.rados.rados.0.trial039.stdout:210: done (3 left) 2026-02-25T08:42:25.273 INFO:tasks.rados.rados.0.trial039.stdout:211: done (2 left) 2026-02-25T08:42:25.273 INFO:tasks.rados.rados.0.trial039.stdout:212: done (1 left) 2026-02-25T08:42:25.273 INFO:tasks.rados.rados.0.trial039.stdout:213: done (0 left) 2026-02-25T08:42:25.273 INFO:tasks.rados.rados.0.trial039.stdout:214: read oid 24 snap 17 2026-02-25T08:42:25.273 INFO:tasks.rados.rados.0.trial039.stdout:214: expect (ObjNum 97 snap 11 seq_num 97) 2026-02-25T08:42:25.273 INFO:tasks.rados.rados.0.trial039.stdout:215: write (excl) oid 44 current snap is 19 2026-02-25T08:42:25.273 INFO:tasks.rados.rados.0.trial039.stdout:215: seq_num 127 ranges {534489=618271,1565566=568923} 2026-02-25T08:42:25.277 INFO:tasks.rados.rados.0.trial039.stdout:215: writing trial03942481-44 from 534489 to 1152760 tid 1 2026-02-25T08:42:25.281 INFO:tasks.rados.rados.0.trial039.stdout:215: writing trial03942481-44 from 1565566 to 2134489 tid 2 2026-02-25T08:42:25.281 INFO:tasks.rados.rados.0.trial039.stdout:216: snap_remove snap 18 2026-02-25T08:42:25.333 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:25.330+0000 7fa0b4393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:25.335 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:25.334+0000 7fa0b4393640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:25.362 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:25.378 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:25.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:26.278 INFO:tasks.rados.rados.0.trial039.stdout:216: done (2 left) 2026-02-25T08:42:26.279 INFO:tasks.rados.rados.0.trial039.stdout:217: read oid 34 snap -1 2026-02-25T08:42:26.279 INFO:tasks.rados.rados.0.trial039.stdout:217: expect (ObjNum 93 snap 11 seq_num 93) 2026-02-25T08:42:26.279 INFO:tasks.rados.rados.0.trial039.stdout:218: read oid 14 snap -1 2026-02-25T08:42:26.279 INFO:tasks.rados.rados.0.trial039.stdout:218: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:42:26.279 INFO:tasks.rados.rados.0.trial039.stdout:219: snap_create 2026-02-25T08:42:26.279 INFO:tasks.rados.rados.0.trial039.stdout:214: expect (ObjNum 97 snap 11 seq_num 97) 2026-02-25T08:42:26.339 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:26.404 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 780501 2026-02-25T08:42:26.460 INFO:tasks.rados.rados.0.trial039.stdout:215: finishing write tid 1 to trial03942481-44 2026-02-25T08:42:26.460 INFO:tasks.rados.rados.0.trial039.stdout:215: finishing write tid 2 to trial03942481-44 2026-02-25T08:42:26.460 INFO:tasks.rados.rados.0.trial039.stdout:215: finishing write tid 3 to trial03942481-44 2026-02-25T08:42:26.460 INFO:tasks.rados.rados.0.trial039.stdout:215: finishing write tid 4 to trial03942481-44 2026-02-25T08:42:26.460 INFO:tasks.rados.rados.0.trial039.stdout:215: oid 44 updating version 0 to 51 2026-02-25T08:42:26.460 INFO:tasks.rados.rados.0.trial039.stdout:215: oid 44 updating version 51 to 53 2026-02-25T08:42:26.460 INFO:tasks.rados.rados.0.trial039.stdout:215: oid 44 version 53 is already newer than 53 2026-02-25T08:42:26.460 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 53 (ObjNum 127 snap 19 seq_num 127) dirty exists 2026-02-25T08:42:26.460 INFO:tasks.rados.rados.0.trial039.stdout:215: left oid 44 (ObjNum 127 snap 19 seq_num 127) 2026-02-25T08:42:26.460 INFO:tasks.rados.rados.0.trial039.stdout:217: expect (ObjNum 93 snap 11 seq_num 93) 2026-02-25T08:42:26.544 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:26.584 INFO:tasks.rados.rados.0.trial039.stdout:218: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:42:26.585 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:26.640 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:27.283 INFO:tasks.rados.rados.0.trial039.stdout:214: done (4 left) 2026-02-25T08:42:27.283 INFO:tasks.rados.rados.0.trial039.stdout:215: done (3 left) 2026-02-25T08:42:27.283 INFO:tasks.rados.rados.0.trial039.stdout:217: done (2 left) 2026-02-25T08:42:27.283 INFO:tasks.rados.rados.0.trial039.stdout:218: done (1 left) 2026-02-25T08:42:27.283 INFO:tasks.rados.rados.0.trial039.stdout:219: done (0 left) 2026-02-25T08:42:27.284 INFO:tasks.rados.rados.0.trial039.stdout:220: write oid 32 current snap is 20 2026-02-25T08:42:27.284 INFO:tasks.rados.rados.0.trial039.stdout:220: seq_num 128 ranges {551296=693520,2024048=127248} 2026-02-25T08:42:27.288 INFO:tasks.rados.rados.0.trial039.stdout:220: writing trial03942481-32 from 551296 to 1244816 tid 1 2026-02-25T08:42:27.292 INFO:tasks.rados.rados.0.trial039.stdout:220: writing trial03942481-32 from 2024048 to 2151296 tid 2 2026-02-25T08:42:27.292 INFO:tasks.rados.rados.0.trial039.stdout:221: read oid 36 snap 16 2026-02-25T08:42:27.292 INFO:tasks.rados.rados.0.trial039.stdout:221: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:42:27.292 INFO:tasks.rados.rados.0.trial039.stdout:222: read oid 4 snap -1 2026-02-25T08:42:27.292 INFO:tasks.rados.rados.0.trial039.stdout:222: expect (ObjNum 94 snap 11 seq_num 94) 2026-02-25T08:42:27.292 INFO:tasks.rados.rados.0.trial039.stdout:223: read oid 13 snap -1 2026-02-25T08:42:27.292 INFO:tasks.rados.rados.0.trial039.stdout:223: expect deleted 2026-02-25T08:42:27.293 INFO:tasks.rados.rados.0.trial039.stdout:224: snap_create 2026-02-25T08:42:27.315 INFO:tasks.rados.rados.0.trial039.stdout:222: expect (ObjNum 94 snap 11 seq_num 94) 2026-02-25T08:42:27.315 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:27.357 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:27.378 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:27.426 INFO:tasks.rados.rados.0.trial039.stdout:220: finishing write tid 1 to trial03942481-32 2026-02-25T08:42:27.426 INFO:tasks.rados.rados.0.trial039.stdout:220: finishing write tid 2 to trial03942481-32 2026-02-25T08:42:27.426 INFO:tasks.rados.rados.0.trial039.stdout:221: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:42:27.456 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:27.454+0000 7ff9825fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:27.458 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:27.457+0000 7ff9825fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:27.485 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:27.487 INFO:tasks.rados.rados.0.trial039.stdout:220: finishing write tid 3 to trial03942481-32 2026-02-25T08:42:27.487 INFO:tasks.rados.rados.0.trial039.stdout:220: finishing write tid 4 to trial03942481-32 2026-02-25T08:42:27.487 INFO:tasks.rados.rados.0.trial039.stdout:220: oid 32 updating version 0 to 37 2026-02-25T08:42:27.487 INFO:tasks.rados.rados.0.trial039.stdout:220: oid 32 version 37 is already newer than 37 2026-02-25T08:42:27.487 INFO:tasks.rados.rados.0.trial039.stdout:220: oid 32 version 37 is already newer than 37 2026-02-25T08:42:27.487 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 32 v 37 (ObjNum 128 snap 20 seq_num 128) dirty exists 2026-02-25T08:42:27.488 INFO:tasks.rados.rados.0.trial039.stdout:220: left oid 32 (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:42:27.501 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:27.501 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:28.284 INFO:tasks.rados.rados.0.trial039.stdout:220: done (4 left) 2026-02-25T08:42:28.284 INFO:tasks.rados.rados.0.trial039.stdout:221: done (3 left) 2026-02-25T08:42:28.285 INFO:tasks.rados.rados.0.trial039.stdout:222: done (2 left) 2026-02-25T08:42:28.285 INFO:tasks.rados.rados.0.trial039.stdout:223: done (1 left) 2026-02-25T08:42:28.285 INFO:tasks.rados.rados.0.trial039.stdout:224: done (0 left) 2026-02-25T08:42:28.285 INFO:tasks.rados.rados.0.trial039.stdout:225: write (excl) oid 4 current snap is 21 2026-02-25T08:42:28.285 INFO:tasks.rados.rados.0.trial039.stdout:225: seq_num 129 ranges {568103=768769,2082530=85573} 2026-02-25T08:42:28.290 INFO:tasks.rados.rados.0.trial039.stdout:225: writing trial03942481-4 from 568103 to 1336872 tid 1 2026-02-25T08:42:28.293 INFO:tasks.rados.rados.0.trial039.stdout:225: writing trial03942481-4 from 2082530 to 2168103 tid 2 2026-02-25T08:42:28.293 INFO:tasks.rados.rados.0.trial039.stdout:226: write oid 37 current snap is 21 2026-02-25T08:42:28.293 INFO:tasks.rados.rados.0.trial039.stdout:226: seq_num 130 ranges {584910=560371,1857365=327545} 2026-02-25T08:42:28.297 INFO:tasks.rados.rados.0.trial039.stdout:226: writing trial03942481-37 from 584910 to 1145281 tid 1 2026-02-25T08:42:28.301 INFO:tasks.rados.rados.0.trial039.stdout:226: writing trial03942481-37 from 1857365 to 2184910 tid 2 2026-02-25T08:42:28.301 INFO:tasks.rados.rados.0.trial039.stdout:227: delete oid 35 current snap is 21 2026-02-25T08:42:28.323 INFO:tasks.rados.rados.0.trial039.stdout:226: finishing write tid 1 to trial03942481-37 2026-02-25T08:42:28.332 INFO:tasks.rados.rados.0.trial039.stdout:226: finishing write tid 2 to trial03942481-37 2026-02-25T08:42:28.334 INFO:tasks.rados.rados.0.trial039.stdout:227: done (2 left) 2026-02-25T08:42:28.334 INFO:tasks.rados.rados.0.trial039.stdout:228: delete oid 31 current snap is 21 2026-02-25T08:42:28.338 INFO:tasks.rados.rados.0.trial039.stdout:226: finishing write tid 3 to trial03942481-37 2026-02-25T08:42:28.338 INFO:tasks.rados.rados.0.trial039.stdout:225: finishing write tid 1 to trial03942481-4 2026-02-25T08:42:28.339 INFO:tasks.rados.rados.0.trial039.stdout:226: finishing write tid 4 to trial03942481-37 2026-02-25T08:42:28.339 INFO:tasks.rados.rados.0.trial039.stdout:226: oid 37 updating version 0 to 56 2026-02-25T08:42:28.339 INFO:tasks.rados.rados.0.trial039.stdout:226: oid 37 version 56 is already newer than 56 2026-02-25T08:42:28.340 INFO:tasks.rados.rados.0.trial039.stdout:226: oid 37 version 56 is already newer than 56 2026-02-25T08:42:28.340 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 37 v 56 (ObjNum 130 snap 21 seq_num 130) dirty exists 2026-02-25T08:42:28.340 INFO:tasks.rados.rados.0.trial039.stdout:226: left oid 37 (ObjNum 130 snap 21 seq_num 130) 2026-02-25T08:42:28.356 INFO:tasks.rados.rados.0.trial039.stdout:225: finishing write tid 2 to trial03942481-4 2026-02-25T08:42:28.368 INFO:tasks.rados.rados.0.trial039.stdout:225: finishing write tid 3 to trial03942481-4 2026-02-25T08:42:28.370 INFO:tasks.rados.rados.0.trial039.stdout:225: finishing write tid 4 to trial03942481-4 2026-02-25T08:42:28.370 INFO:tasks.rados.rados.0.trial039.stdout:225: oid 4 updating version 0 to 34 2026-02-25T08:42:28.370 INFO:tasks.rados.rados.0.trial039.stdout:225: oid 4 version 34 is already newer than 34 2026-02-25T08:42:28.370 INFO:tasks.rados.rados.0.trial039.stdout:225: oid 4 version 34 is already newer than 32 2026-02-25T08:42:28.370 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 34 (ObjNum 129 snap 21 seq_num 129) dirty exists 2026-02-25T08:42:28.371 INFO:tasks.rados.rados.0.trial039.stdout:225: left oid 4 (ObjNum 129 snap 21 seq_num 129) 2026-02-25T08:42:28.384 INFO:tasks.rados.rados.0.trial039.stdout:225: done (2 left) 2026-02-25T08:42:28.384 INFO:tasks.rados.rados.0.trial039.stdout:226: done (1 left) 2026-02-25T08:42:28.384 INFO:tasks.rados.rados.0.trial039.stdout:228: done (0 left) 2026-02-25T08:42:28.384 INFO:tasks.rados.rados.0.trial039.stdout:229: delete oid 46 current snap is 21 2026-02-25T08:42:28.392 INFO:tasks.rados.rados.0.trial039.stdout:229: done (0 left) 2026-02-25T08:42:28.392 INFO:tasks.rados.rados.0.trial039.stdout:230: write oid 26 current snap is 21 2026-02-25T08:42:28.392 INFO:tasks.rados.rados.0.trial039.stdout:230: seq_num 134 ranges {652138=461367,1574940=677198} 2026-02-25T08:42:28.395 INFO:tasks.rados.rados.0.trial039.stdout:230: writing trial03942481-26 from 652138 to 1113505 tid 1 2026-02-25T08:42:28.400 INFO:tasks.rados.rados.0.trial039.stdout:230: writing trial03942481-26 from 1574940 to 2252138 tid 2 2026-02-25T08:42:28.400 INFO:tasks.rados.rados.0.trial039.stdout:231: snap_remove snap 17 2026-02-25T08:42:29.289 INFO:tasks.rados.rados.0.trial039.stdout:231: done (1 left) 2026-02-25T08:42:29.290 INFO:tasks.rados.rados.0.trial039.stdout:232: snap_create 2026-02-25T08:42:29.290 INFO:tasks.rados.rados.0.trial039.stdout:230: finishing write tid 1 to trial03942481-26 2026-02-25T08:42:29.290 INFO:tasks.rados.rados.0.trial039.stdout:230: finishing write tid 2 to trial03942481-26 2026-02-25T08:42:29.290 INFO:tasks.rados.rados.0.trial039.stdout:230: finishing write tid 3 to trial03942481-26 2026-02-25T08:42:29.290 INFO:tasks.rados.rados.0.trial039.stdout:230: finishing write tid 4 to trial03942481-26 2026-02-25T08:42:29.290 INFO:tasks.rados.rados.0.trial039.stdout:230: oid 26 updating version 0 to 26 2026-02-25T08:42:29.290 INFO:tasks.rados.rados.0.trial039.stdout:230: oid 26 version 26 is already newer than 26 2026-02-25T08:42:29.290 INFO:tasks.rados.rados.0.trial039.stdout:230: oid 26 version 26 is already newer than 26 2026-02-25T08:42:29.290 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 26 v 26 (ObjNum 134 snap 21 seq_num 134) dirty exists 2026-02-25T08:42:29.291 INFO:tasks.rados.rados.0.trial039.stdout:230: left oid 26 (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:42:29.502 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:29.581 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:29.578+0000 7fa3a650b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:29.583 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:29.581+0000 7fa3a650b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:29.597 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:29.610 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:29.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:29.671 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:29.674 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:30.293 INFO:tasks.rados.rados.0.trial039.stdout:230: done (1 left) 2026-02-25T08:42:30.293 INFO:tasks.rados.rados.0.trial039.stdout:232: done (0 left) 2026-02-25T08:42:30.294 INFO:tasks.rados.rados.0.trial039.stdout:233: write (excl) oid 29 current snap is 22 2026-02-25T08:42:30.294 INFO:tasks.rados.rados.0.trial039.stdout:233: seq_num 135 ranges {668945=536616,1633422=491363} 2026-02-25T08:42:30.297 INFO:tasks.rados.rados.0.trial039.stdout:233: writing trial03942481-29 from 668945 to 1205561 tid 1 2026-02-25T08:42:30.301 INFO:tasks.rados.rados.0.trial039.stdout:233: writing trial03942481-29 from 1633422 to 2124785 tid 2 2026-02-25T08:42:30.301 INFO:tasks.rados.rados.0.trial039.stdout:234: snap_remove snap 21 2026-02-25T08:42:30.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~55s 2026-02-25T08:42:31.298 INFO:tasks.rados.rados.0.trial039.stdout:234: done (1 left) 2026-02-25T08:42:31.299 INFO:tasks.rados.rados.0.trial039.stdout:235: write (excl) oid 28 current snap is 22 2026-02-25T08:42:31.299 INFO:tasks.rados.rados.0.trial039.stdout:233: finishing write tid 1 to trial03942481-29 2026-02-25T08:42:31.299 INFO:tasks.rados.rados.0.trial039.stdout:235: seq_num 136 ranges {685752=611865,1975551=310201} 2026-02-25T08:42:31.303 INFO:tasks.rados.rados.0.trial039.stdout:235: writing trial03942481-28 from 685752 to 1297617 tid 1 2026-02-25T08:42:31.307 INFO:tasks.rados.rados.0.trial039.stdout:235: writing trial03942481-28 from 1975551 to 2285752 tid 2 2026-02-25T08:42:31.307 INFO:tasks.rados.rados.0.trial039.stdout:236: write oid 19 current snap is 22 2026-02-25T08:42:31.307 INFO:tasks.rados.rados.0.trial039.stdout:233: finishing write tid 2 to trial03942481-29 2026-02-25T08:42:31.307 INFO:tasks.rados.rados.0.trial039.stdout:233: finishing write tid 3 to trial03942481-29 2026-02-25T08:42:31.308 INFO:tasks.rados.rados.0.trial039.stdout:236: seq_num 137 ranges {702559=403467,1750386=495032} 2026-02-25T08:42:31.311 INFO:tasks.rados.rados.0.trial039.stdout:236: writing trial03942481-19 from 702559 to 1106026 tid 1 2026-02-25T08:42:31.317 INFO:tasks.rados.rados.0.trial039.stdout:236: writing trial03942481-19 from 1750386 to 2245418 tid 2 2026-02-25T08:42:31.317 INFO:tasks.rados.rados.0.trial039.stdout:237: read oid 41 snap 20 2026-02-25T08:42:31.317 INFO:tasks.rados.rados.0.trial039.stdout:237: expect (ObjNum 65 snap 4 seq_num 65) 2026-02-25T08:42:31.317 INFO:tasks.rados.rados.0.trial039.stdout:238: read oid 20 snap -1 2026-02-25T08:42:31.317 INFO:tasks.rados.rados.0.trial039.stdout:238: expect (ObjNum 116 snap 14 seq_num 116) 2026-02-25T08:42:31.318 INFO:tasks.rados.rados.0.trial039.stdout:239: snap_remove snap 19 2026-02-25T08:42:31.628 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:31.703 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:31.701+0000 7ff0a0635640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:31.705 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:31.704+0000 7ff0a0635640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:31.731 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:31.748 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:32.303 INFO:tasks.rados.rados.0.trial039.stdout:239: done (5 left) 2026-02-25T08:42:32.303 INFO:tasks.rados.rados.0.trial039.stdout:240: read oid 11 snap -1 2026-02-25T08:42:32.304 INFO:tasks.rados.rados.0.trial039.stdout:240: expect (ObjNum 122 snap 17 seq_num 122) 2026-02-25T08:42:32.304 INFO:tasks.rados.rados.0.trial039.stdout:241: delete oid 36 current snap is 22 2026-02-25T08:42:32.304 INFO:tasks.rados.rados.0.trial039.stdout:233: finishing write tid 4 to trial03942481-29 2026-02-25T08:42:32.304 INFO:tasks.rados.rados.0.trial039.stdout:233: oid 29 updating version 0 to 30 2026-02-25T08:42:32.304 INFO:tasks.rados.rados.0.trial039.stdout:233: oid 29 version 30 is already newer than 30 2026-02-25T08:42:32.305 INFO:tasks.rados.rados.0.trial039.stdout:233: oid 29 version 30 is already newer than 28 2026-02-25T08:42:32.305 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 29 v 30 (ObjNum 135 snap 22 seq_num 135) dirty exists 2026-02-25T08:42:32.305 INFO:tasks.rados.rados.0.trial039.stdout:233: left oid 29 (ObjNum 135 snap 22 seq_num 135) 2026-02-25T08:42:32.305 INFO:tasks.rados.rados.0.trial039.stdout:237: expect (ObjNum 65 snap 4 seq_num 65) 2026-02-25T08:42:32.339 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:32.389 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2381054 2026-02-25T08:42:32.400 INFO:tasks.rados.rados.0.trial039.stdout:235: finishing write tid 1 to trial03942481-28 2026-02-25T08:42:32.400 INFO:tasks.rados.rados.0.trial039.stdout:236: finishing write tid 1 to trial03942481-19 2026-02-25T08:42:32.400 INFO:tasks.rados.rados.0.trial039.stdout:236: finishing write tid 2 to trial03942481-19 2026-02-25T08:42:32.400 INFO:tasks.rados.rados.0.trial039.stdout:235: finishing write tid 2 to trial03942481-28 2026-02-25T08:42:32.400 INFO:tasks.rados.rados.0.trial039.stdout:238: expect (ObjNum 116 snap 14 seq_num 116) 2026-02-25T08:42:32.454 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 4085 2026-02-25T08:42:32.567 INFO:tasks.rados.rados.0.trial039.stdout:235: finishing write tid 3 to trial03942481-28 2026-02-25T08:42:32.568 INFO:tasks.rados.rados.0.trial039.stdout:236: finishing write tid 3 to trial03942481-19 2026-02-25T08:42:32.568 INFO:tasks.rados.rados.0.trial039.stdout:233: done (6 left) 2026-02-25T08:42:32.568 INFO:tasks.rados.rados.0.trial039.stdout:237: done (5 left) 2026-02-25T08:42:32.568 INFO:tasks.rados.rados.0.trial039.stdout:238: done (4 left) 2026-02-25T08:42:32.568 INFO:tasks.rados.rados.0.trial039.stdout:241: done (3 left) 2026-02-25T08:42:32.568 INFO:tasks.rados.rados.0.trial039.stdout:242: write (excl) oid 35 current snap is 22 2026-02-25T08:42:32.568 INFO:tasks.rados.rados.0.trial039.stdout:236: finishing write tid 4 to trial03942481-19 2026-02-25T08:42:32.568 INFO:tasks.rados.rados.0.trial039.stdout:236: oid 19 updating version 0 to 83 2026-02-25T08:42:32.568 INFO:tasks.rados.rados.0.trial039.stdout:236: oid 19 version 83 is already newer than 83 2026-02-25T08:42:32.569 INFO:tasks.rados.rados.0.trial039.stdout:236: oid 19 version 83 is already newer than 83 2026-02-25T08:42:32.569 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 19 v 83 (ObjNum 137 snap 22 seq_num 137) dirty exists 2026-02-25T08:42:32.569 INFO:tasks.rados.rados.0.trial039.stdout:236: left oid 19 (ObjNum 137 snap 22 seq_num 137) 2026-02-25T08:42:32.569 INFO:tasks.rados.rados.0.trial039.stdout:235: finishing write tid 4 to trial03942481-28 2026-02-25T08:42:32.569 INFO:tasks.rados.rados.0.trial039.stdout:235: oid 28 updating version 0 to 35 2026-02-25T08:42:32.569 INFO:tasks.rados.rados.0.trial039.stdout:235: oid 28 version 35 is already newer than 33 2026-02-25T08:42:32.569 INFO:tasks.rados.rados.0.trial039.stdout:235: oid 28 version 35 is already newer than 35 2026-02-25T08:42:32.569 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 28 v 35 (ObjNum 136 snap 22 seq_num 136) dirty exists 2026-02-25T08:42:32.569 INFO:tasks.rados.rados.0.trial039.stdout:235: left oid 28 (ObjNum 136 snap 22 seq_num 136) 2026-02-25T08:42:32.569 INFO:tasks.rados.rados.0.trial039.stdout:242: seq_num 139 ranges {736173=553965,1867350=468823} 2026-02-25T08:42:32.572 INFO:tasks.rados.rados.0.trial039.stdout:242: writing trial03942481-35 from 736173 to 1290138 tid 1 2026-02-25T08:42:32.576 INFO:tasks.rados.rados.0.trial039.stdout:242: writing trial03942481-35 from 1867350 to 2336173 tid 2 2026-02-25T08:42:32.576 INFO:tasks.rados.rados.0.trial039.stdout:235: done (3 left) 2026-02-25T08:42:32.576 INFO:tasks.rados.rados.0.trial039.stdout:236: done (2 left) 2026-02-25T08:42:32.576 INFO:tasks.rados.rados.0.trial039.stdout:243: snap_create 2026-02-25T08:42:32.576 INFO:tasks.rados.rados.0.trial039.stdout:240: expect (ObjNum 122 snap 17 seq_num 122) 2026-02-25T08:42:32.576 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:32.598 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:32.636 INFO:tasks.rados.rados.0.trial039.stdout:242: finishing write tid 1 to trial03942481-35 2026-02-25T08:42:32.636 INFO:tasks.rados.rados.0.trial039.stdout:242: finishing write tid 2 to trial03942481-35 2026-02-25T08:42:32.636 INFO:tasks.rados.rados.0.trial039.stdout:242: finishing write tid 3 to trial03942481-35 2026-02-25T08:42:32.637 INFO:tasks.rados.rados.0.trial039.stdout:242: finishing write tid 4 to trial03942481-35 2026-02-25T08:42:32.637 INFO:tasks.rados.rados.0.trial039.stdout:242: oid 35 updating version 0 to 27 2026-02-25T08:42:32.637 INFO:tasks.rados.rados.0.trial039.stdout:242: oid 35 version 27 is already newer than 27 2026-02-25T08:42:32.637 INFO:tasks.rados.rados.0.trial039.stdout:242: oid 35 version 27 is already newer than 25 2026-02-25T08:42:32.637 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 35 v 27 (ObjNum 139 snap 22 seq_num 139) dirty exists 2026-02-25T08:42:32.637 INFO:tasks.rados.rados.0.trial039.stdout:242: left oid 35 (ObjNum 139 snap 22 seq_num 139) 2026-02-25T08:42:33.051 INFO:tasks.rados.rados.0.trial039.stdout:240: done (2 left) 2026-02-25T08:42:33.051 INFO:tasks.rados.rados.0.trial039.stdout:242: done (1 left) 2026-02-25T08:42:33.051 INFO:tasks.rados.rados.0.trial039.stdout:243: done (0 left) 2026-02-25T08:42:33.051 INFO:tasks.rados.rados.0.trial039.stdout:244: write (excl) oid 13 current snap is 23 2026-02-25T08:42:33.051 INFO:tasks.rados.rados.0.trial039.stdout:244: seq_num 140 ranges {752980=629214,1809479=543501} 2026-02-25T08:42:33.056 INFO:tasks.rados.rados.0.trial039.stdout:244: writing trial03942481-13 from 752980 to 1382194 tid 1 2026-02-25T08:42:33.060 INFO:tasks.rados.rados.0.trial039.stdout:244: writing trial03942481-13 from 1809479 to 2352980 tid 2 2026-02-25T08:42:33.060 INFO:tasks.rados.rados.0.trial039.stdout:245: snap_create 2026-02-25T08:42:33.078 INFO:tasks.rados.rados.0.trial039.stdout:244: finishing write tid 1 to trial03942481-13 2026-02-25T08:42:33.089 INFO:tasks.rados.rados.0.trial039.stdout:244: finishing write tid 2 to trial03942481-13 2026-02-25T08:42:33.093 INFO:tasks.rados.rados.0.trial039.stdout:244: finishing write tid 3 to trial03942481-13 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial039.stdout:244: finishing write tid 4 to trial03942481-13 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial039.stdout:244: oid 13 updating version 0 to 37 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial039.stdout:244: oid 13 version 37 is already newer than 35 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial039.stdout:244: oid 13 version 37 is already newer than 37 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 13 v 37 (ObjNum 140 snap 23 seq_num 140) dirty exists 2026-02-25T08:42:33.095 INFO:tasks.rados.rados.0.trial039.stdout:244: left oid 13 (ObjNum 140 snap 23 seq_num 140) 2026-02-25T08:42:33.748 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:33.827 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:33.824+0000 7f888df6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:33.829 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:33.828+0000 7f888df6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:33.856 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:33.873 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:34.052 INFO:tasks.rados.rados.0.trial039.stdout:244: done (1 left) 2026-02-25T08:42:34.052 INFO:tasks.rados.rados.0.trial039.stdout:245: done (0 left) 2026-02-25T08:42:34.052 INFO:tasks.rados.rados.0.trial039.stdout:246: read oid 49 snap -1 2026-02-25T08:42:34.052 INFO:tasks.rados.rados.0.trial039.stdout:246: expect (ObjNum 108 snap 12 seq_num 108) 2026-02-25T08:42:34.053 INFO:tasks.rados.rados.0.trial039.stdout:247: read oid 9 snap -1 2026-02-25T08:42:34.053 INFO:tasks.rados.rados.0.trial039.stdout:247: expect (ObjNum 117 snap 16 seq_num 117) 2026-02-25T08:42:34.053 INFO:tasks.rados.rados.0.trial039.stdout:248: snap_remove snap 22 2026-02-25T08:42:34.675 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:34.746 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:34.749 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:34.750 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:35.052 INFO:tasks.rados.rados.0.trial039.stdout:248: done (2 left) 2026-02-25T08:42:35.053 INFO:tasks.rados.rados.0.trial039.stdout:249: delete oid 33 current snap is 24 2026-02-25T08:42:35.053 INFO:tasks.rados.rados.0.trial039.stdout:247: expect (ObjNum 117 snap 16 seq_num 117) 2026-02-25T08:42:35.053 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:35.144 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 648902 2026-02-25T08:42:35.188 INFO:tasks.rados.rados.0.trial039.stdout:246: expect (ObjNum 108 snap 12 seq_num 108) 2026-02-25T08:42:35.188 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:35.234 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:35.303 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 3485477 2026-02-25T08:42:35.310 INFO:tasks.rados.rados.0.trial039.stdout:246: done (2 left) 2026-02-25T08:42:35.310 INFO:tasks.rados.rados.0.trial039.stdout:247: done (1 left) 2026-02-25T08:42:35.310 INFO:tasks.rados.rados.0.trial039.stdout:249: done (0 left) 2026-02-25T08:42:35.310 INFO:tasks.rados.rados.0.trial039.stdout:250: write (excl) oid 24 current snap is 24 2026-02-25T08:42:35.311 INFO:tasks.rados.rados.0.trial039.stdout:250: seq_num 142 ranges {786594=779712,2326443=60151} 2026-02-25T08:42:35.316 INFO:tasks.rados.rados.0.trial039.stdout:250: writing trial03942481-24 from 786594 to 1566306 tid 1 2026-02-25T08:42:35.320 INFO:tasks.rados.rados.0.trial039.stdout:250: writing trial03942481-24 from 2326443 to 2386594 tid 2 2026-02-25T08:42:35.321 INFO:tasks.rados.rados.0.trial039.stdout:251: read oid 7 snap -1 2026-02-25T08:42:35.321 INFO:tasks.rados.rados.0.trial039.stdout:251: expect (ObjNum 109 snap 13 seq_num 109) 2026-02-25T08:42:35.321 INFO:tasks.rados.rados.0.trial039.stdout:252: read oid 45 snap -1 2026-02-25T08:42:35.321 INFO:tasks.rados.rados.0.trial039.stdout:252: expect (ObjNum 114 snap 14 seq_num 114) 2026-02-25T08:42:35.321 INFO:tasks.rados.rados.0.trial039.stdout:253: snap_remove snap 23 2026-02-25T08:42:35.874 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:35.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~61s 2026-02-25T08:42:35.951 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:35.948+0000 7f7a9bc8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:35.953 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:35.952+0000 7f7a9aa02640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:35.979 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:35.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:36.309 INFO:tasks.rados.rados.0.trial039.stdout:253: done (3 left) 2026-02-25T08:42:36.309 INFO:tasks.rados.rados.0.trial039.stdout:254: delete oid 36 current snap is 24 2026-02-25T08:42:36.309 INFO:tasks.rados.rados.0.trial039.stdout:251: expect (ObjNum 109 snap 13 seq_num 109) 2026-02-25T08:42:36.309 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:36.365 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:36.425 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1204389 2026-02-25T08:42:36.467 INFO:tasks.rados.rados.0.trial039.stdout:252: expect (ObjNum 114 snap 14 seq_num 114) 2026-02-25T08:42:36.625 INFO:tasks.rados.rados.0.trial039.stdout:250: finishing write tid 1 to trial03942481-24 2026-02-25T08:42:36.625 INFO:tasks.rados.rados.0.trial039.stdout:250: finishing write tid 2 to trial03942481-24 2026-02-25T08:42:36.625 INFO:tasks.rados.rados.0.trial039.stdout:251: done (3 left) 2026-02-25T08:42:36.625 INFO:tasks.rados.rados.0.trial039.stdout:252: done (2 left) 2026-02-25T08:42:36.626 INFO:tasks.rados.rados.0.trial039.stdout:254: done (1 left) 2026-02-25T08:42:36.626 INFO:tasks.rados.rados.0.trial039.stdout:255: snap_create 2026-02-25T08:42:36.626 INFO:tasks.rados.rados.0.trial039.stdout:250: finishing write tid 3 to trial03942481-24 2026-02-25T08:42:36.626 INFO:tasks.rados.rados.0.trial039.stdout:250: finishing write tid 4 to trial03942481-24 2026-02-25T08:42:36.626 INFO:tasks.rados.rados.0.trial039.stdout:250: oid 24 updating version 0 to 29 2026-02-25T08:42:36.626 INFO:tasks.rados.rados.0.trial039.stdout:250: oid 24 version 29 is already newer than 29 2026-02-25T08:42:36.626 INFO:tasks.rados.rados.0.trial039.stdout:250: oid 24 version 29 is already newer than 27 2026-02-25T08:42:36.626 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 24 v 29 (ObjNum 142 snap 24 seq_num 142) dirty exists 2026-02-25T08:42:36.626 INFO:tasks.rados.rados.0.trial039.stdout:250: left oid 24 (ObjNum 142 snap 24 seq_num 142) 2026-02-25T08:42:37.313 INFO:tasks.rados.rados.0.trial039.stdout:250: done (1 left) 2026-02-25T08:42:37.313 INFO:tasks.rados.rados.0.trial039.stdout:255: done (0 left) 2026-02-25T08:42:37.313 INFO:tasks.rados.rados.0.trial039.stdout:256: snap_create 2026-02-25T08:42:37.997 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:38.074 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:38.072+0000 7f6f33bfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:38.077 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:38.075+0000 7f6f33bfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:38.102 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:38.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:38.312 INFO:tasks.rados.rados.0.trial039.stdout:256: done (0 left) 2026-02-25T08:42:38.312 INFO:tasks.rados.rados.0.trial039.stdout:257: snap_remove snap 20 2026-02-25T08:42:39.316 INFO:tasks.rados.rados.0.trial039.stdout:257: done (0 left) 2026-02-25T08:42:39.316 INFO:tasks.rados.rados.0.trial039.stdout:258: snap_create 2026-02-25T08:42:39.751 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:39.824 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:39.827 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:40.120 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:40.196 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:40.194+0000 7f043b78c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:40.198 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:40.197+0000 7f043b78c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:40.223 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:40.241 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:40.241 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:40.320 INFO:tasks.rados.rados.0.trial039.stdout:258: done (0 left) 2026-02-25T08:42:40.320 INFO:tasks.rados.rados.0.trial039.stdout:259: write (excl) oid 4 current snap is 27 2026-02-25T08:42:40.320 INFO:tasks.rados.rados.0.trial039.stdout:259: seq_num 144 ranges {420208=530210,1527054=766050} 2026-02-25T08:42:40.323 INFO:tasks.rados.rados.0.trial039.stdout:259: writing trial03942481-4 from 420208 to 950418 tid 1 2026-02-25T08:42:40.329 INFO:tasks.rados.rados.0.trial039.stdout:259: writing trial03942481-4 from 1527054 to 2293104 tid 2 2026-02-25T08:42:40.330 INFO:tasks.rados.rados.0.trial039.stdout:260: write (excl) oid 22 current snap is 27 2026-02-25T08:42:40.330 INFO:tasks.rados.rados.0.trial039.stdout:260: seq_num 145 ranges {437015=721812,1701889=509708} 2026-02-25T08:42:40.334 INFO:tasks.rados.rados.0.trial039.stdout:260: writing trial03942481-22 from 437015 to 1158827 tid 1 2026-02-25T08:42:40.338 INFO:tasks.rados.rados.0.trial039.stdout:260: writing trial03942481-22 from 1701889 to 2211597 tid 2 2026-02-25T08:42:40.339 INFO:tasks.rados.rados.0.trial039.stdout:261: write oid 34 current snap is 27 2026-02-25T08:42:40.339 INFO:tasks.rados.rados.0.trial039.stdout:261: seq_num 146 ranges {453822=797061,1760371=653366} 2026-02-25T08:42:40.344 INFO:tasks.rados.rados.0.trial039.stdout:261: writing trial03942481-34 from 453822 to 1250883 tid 1 2026-02-25T08:42:40.349 INFO:tasks.rados.rados.0.trial039.stdout:261: writing trial03942481-34 from 1760371 to 2413737 tid 2 2026-02-25T08:42:40.349 INFO:tasks.rados.rados.0.trial039.stdout:262: write oid 15 current snap is 27 2026-02-25T08:42:40.349 INFO:tasks.rados.rados.0.trial039.stdout:262: seq_num 147 ranges {470629=472310,1418853=513377} 2026-02-25T08:42:40.353 INFO:tasks.rados.rados.0.trial039.stdout:262: writing trial03942481-15 from 470629 to 942939 tid 1 2026-02-25T08:42:40.357 INFO:tasks.rados.rados.0.trial039.stdout:262: writing trial03942481-15 from 1418853 to 1932230 tid 2 2026-02-25T08:42:40.357 INFO:tasks.rados.rados.0.trial039.stdout:263: read oid 10 snap 25 2026-02-25T08:42:40.357 INFO:tasks.rados.rados.0.trial039.stdout:263: expect (ObjNum 87 snap 9 seq_num 87) 2026-02-25T08:42:40.357 INFO:tasks.rados.rados.0.trial039.stdout:264: read oid 1 snap -1 2026-02-25T08:42:40.357 INFO:tasks.rados.rados.0.trial039.stdout:264: expect (ObjNum 102 snap 11 seq_num 102) 2026-02-25T08:42:40.358 INFO:tasks.rados.rados.0.trial039.stdout:265: delete oid 29 current snap is 27 2026-02-25T08:42:40.361 INFO:tasks.rados.rados.0.trial039.stdout:260: finishing write tid 1 to trial03942481-22 2026-02-25T08:42:40.371 INFO:tasks.rados.rados.0.trial039.stdout:259: finishing write tid 1 to trial03942481-4 2026-02-25T08:42:40.392 INFO:tasks.rados.rados.0.trial039.stdout:260: finishing write tid 2 to trial03942481-22 2026-02-25T08:42:40.398 INFO:tasks.rados.rados.0.trial039.stdout:259: finishing write tid 2 to trial03942481-4 2026-02-25T08:42:40.399 INFO:tasks.rados.rados.0.trial039.stdout:263: expect (ObjNum 87 snap 9 seq_num 87) 2026-02-25T08:42:40.405 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 848101 2026-02-25T08:42:40.439 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 868276 2026-02-25T08:42:40.503 INFO:tasks.rados.rados.0.trial039.stdout:263: done (6 left) 2026-02-25T08:42:40.503 INFO:tasks.rados.rados.0.trial039.stdout:265: done (5 left) 2026-02-25T08:42:40.503 INFO:tasks.rados.rados.0.trial039.stdout:266: snap_remove snap 24 2026-02-25T08:42:41.326 INFO:tasks.rados.rados.0.trial039.stdout:266: done (5 left) 2026-02-25T08:42:41.326 INFO:tasks.rados.rados.0.trial039.stdout:267: write oid 27 current snap is 27 2026-02-25T08:42:41.326 INFO:tasks.rados.rados.0.trial039.stdout:260: finishing write tid 3 to trial03942481-22 2026-02-25T08:42:41.326 INFO:tasks.rados.rados.0.trial039.stdout:267: seq_num 149 ranges {504243=622808,1819464=517046} 2026-02-25T08:42:41.329 INFO:tasks.rados.rados.0.trial039.stdout:267: writing trial03942481-27 from 504243 to 1127051 tid 1 2026-02-25T08:42:41.335 INFO:tasks.rados.rados.0.trial039.stdout:267: writing trial03942481-27 from 1819464 to 2336510 tid 2 2026-02-25T08:42:41.335 INFO:tasks.rados.rados.0.trial039.stdout:268: read oid 14 snap -1 2026-02-25T08:42:41.335 INFO:tasks.rados.rados.0.trial039.stdout:268: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:42:41.335 INFO:tasks.rados.rados.0.trial039.stdout:269: read oid 40 snap 25 2026-02-25T08:42:41.335 INFO:tasks.rados.rados.0.trial039.stdout:269: expect (ObjNum 92 snap 11 seq_num 92) 2026-02-25T08:42:41.335 INFO:tasks.rados.rados.0.trial039.stdout:270: read oid 2 snap -1 2026-02-25T08:42:41.336 INFO:tasks.rados.rados.0.trial039.stdout:270: expect deleted 2026-02-25T08:42:41.336 INFO:tasks.rados.rados.0.trial039.stdout:271: read oid 16 snap -1 2026-02-25T08:42:41.336 INFO:tasks.rados.rados.0.trial039.stdout:271: expect deleted 2026-02-25T08:42:41.336 INFO:tasks.rados.rados.0.trial039.stdout:272: snap_create 2026-02-25T08:42:41.336 INFO:tasks.rados.rados.0.trial039.stdout:264: expect (ObjNum 102 snap 11 seq_num 102) 2026-02-25T08:42:41.376 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:41.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~66s 2026-02-25T08:42:41.439 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 3252429 2026-02-25T08:42:41.448 INFO:tasks.rados.rados.0.trial039.stdout:259: finishing write tid 3 to trial03942481-4 2026-02-25T08:42:41.448 INFO:tasks.rados.rados.0.trial039.stdout:260: finishing write tid 4 to trial03942481-22 2026-02-25T08:42:41.448 INFO:tasks.rados.rados.0.trial039.stdout:260: oid 22 updating version 0 to 12 2026-02-25T08:42:41.448 INFO:tasks.rados.rados.0.trial039.stdout:260: oid 22 version 12 is already newer than 10 2026-02-25T08:42:41.448 INFO:tasks.rados.rados.0.trial039.stdout:260: oid 22 version 12 is already newer than 12 2026-02-25T08:42:41.448 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 22 v 12 (ObjNum 145 snap 27 seq_num 145) dirty exists 2026-02-25T08:42:41.448 INFO:tasks.rados.rados.0.trial039.stdout:260: left oid 22 (ObjNum 145 snap 27 seq_num 145) 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:259: finishing write tid 4 to trial03942481-4 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:259: oid 4 updating version 0 to 39 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:259: oid 4 updating version 39 to 41 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:259: oid 4 version 41 is already newer than 41 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 41 (ObjNum 144 snap 27 seq_num 144) dirty exists 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:259: left oid 4 (ObjNum 144 snap 27 seq_num 144) 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:261: finishing write tid 1 to trial03942481-34 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:262: finishing write tid 1 to trial03942481-15 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:261: finishing write tid 2 to trial03942481-34 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:262: finishing write tid 2 to trial03942481-15 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:261: finishing write tid 3 to trial03942481-34 2026-02-25T08:42:41.449 INFO:tasks.rados.rados.0.trial039.stdout:261: finishing write tid 4 to trial03942481-34 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:261: oid 34 updating version 0 to 63 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:261: oid 34 version 63 is already newer than 63 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:261: oid 34 version 63 is already newer than 63 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 63 (ObjNum 146 snap 27 seq_num 146) dirty exists 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:261: left oid 34 (ObjNum 146 snap 27 seq_num 146) 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:262: finishing write tid 3 to trial03942481-15 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:262: finishing write tid 4 to trial03942481-15 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:262: oid 15 updating version 0 to 45 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:262: oid 15 version 45 is already newer than 45 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:262: oid 15 version 45 is already newer than 45 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 15 v 45 (ObjNum 147 snap 27 seq_num 147) dirty exists 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:262: left oid 15 (ObjNum 147 snap 27 seq_num 147) 2026-02-25T08:42:41.450 INFO:tasks.rados.rados.0.trial039.stdout:268: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:42:41.453 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1231056 2026-02-25T08:42:41.478 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 921600 2026-02-25T08:42:41.503 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:41.536 INFO:tasks.rados.rados.0.trial039.stdout:267: finishing write tid 1 to trial03942481-27 2026-02-25T08:42:41.536 INFO:tasks.rados.rados.0.trial039.stdout:267: finishing write tid 2 to trial03942481-27 2026-02-25T08:42:41.536 INFO:tasks.rados.rados.0.trial039.stdout:269: expect (ObjNum 92 snap 11 seq_num 92) 2026-02-25T08:42:41.572 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:41.629 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2313917 2026-02-25T08:42:41.642 INFO:tasks.rados.rados.0.trial039.stdout:267: finishing write tid 3 to trial03942481-27 2026-02-25T08:42:41.642 INFO:tasks.rados.rados.0.trial039.stdout:267: finishing write tid 4 to trial03942481-27 2026-02-25T08:42:41.642 INFO:tasks.rados.rados.0.trial039.stdout:267: oid 27 updating version 0 to 42 2026-02-25T08:42:41.642 INFO:tasks.rados.rados.0.trial039.stdout:267: oid 27 version 42 is already newer than 42 2026-02-25T08:42:41.642 INFO:tasks.rados.rados.0.trial039.stdout:267: oid 27 version 42 is already newer than 42 2026-02-25T08:42:41.642 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 27 v 42 (ObjNum 149 snap 27 seq_num 149) dirty exists 2026-02-25T08:42:41.642 INFO:tasks.rados.rados.0.trial039.stdout:267: left oid 27 (ObjNum 149 snap 27 seq_num 149) 2026-02-25T08:42:42.242 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:42.320 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:42.317+0000 7f2402e8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:42.323 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:42.321+0000 7f2402e8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:42.327 INFO:tasks.rados.rados.0.trial039.stdout:259: done (10 left) 2026-02-25T08:42:42.327 INFO:tasks.rados.rados.0.trial039.stdout:260: done (9 left) 2026-02-25T08:42:42.327 INFO:tasks.rados.rados.0.trial039.stdout:261: done (8 left) 2026-02-25T08:42:42.327 INFO:tasks.rados.rados.0.trial039.stdout:262: done (7 left) 2026-02-25T08:42:42.328 INFO:tasks.rados.rados.0.trial039.stdout:264: done (6 left) 2026-02-25T08:42:42.328 INFO:tasks.rados.rados.0.trial039.stdout:267: done (5 left) 2026-02-25T08:42:42.328 INFO:tasks.rados.rados.0.trial039.stdout:268: done (4 left) 2026-02-25T08:42:42.328 INFO:tasks.rados.rados.0.trial039.stdout:269: done (3 left) 2026-02-25T08:42:42.328 INFO:tasks.rados.rados.0.trial039.stdout:270: done (2 left) 2026-02-25T08:42:42.328 INFO:tasks.rados.rados.0.trial039.stdout:271: done (1 left) 2026-02-25T08:42:42.328 INFO:tasks.rados.rados.0.trial039.stdout:272: done (0 left) 2026-02-25T08:42:42.328 INFO:tasks.rados.rados.0.trial039.stdout:273: read oid 13 snap 25 2026-02-25T08:42:42.328 INFO:tasks.rados.rados.0.trial039.stdout:273: expect (ObjNum 140 snap 23 seq_num 140) 2026-02-25T08:42:42.328 INFO:tasks.rados.rados.0.trial039.stdout:274: snap_remove snap 16 2026-02-25T08:42:42.361 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:42.376 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:43.052 INFO:tasks.rados.rados.0.trial039.stdout:274: done (1 left) 2026-02-25T08:42:43.053 INFO:tasks.rados.rados.0.trial039.stdout:275: write (excl) oid 36 current snap is 28 2026-02-25T08:42:43.053 INFO:tasks.rados.rados.0.trial039.stdout:275: seq_num 150 ranges {521050=698057,1877946=643104} 2026-02-25T08:42:43.057 INFO:tasks.rados.rados.0.trial039.stdout:275: writing trial03942481-36 from 521050 to 1219107 tid 1 2026-02-25T08:42:43.062 INFO:tasks.rados.rados.0.trial039.stdout:275: writing trial03942481-36 from 1877946 to 2521050 tid 2 2026-02-25T08:42:43.062 INFO:tasks.rados.rados.0.trial039.stdout:276: snap_create 2026-02-25T08:42:43.062 INFO:tasks.rados.rados.0.trial039.stdout:273: expect (ObjNum 140 snap 23 seq_num 140) 2026-02-25T08:42:43.104 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:43.124 INFO:tasks.rados.rados.0.trial039.stdout:275: finishing write tid 1 to trial03942481-36 2026-02-25T08:42:43.124 INFO:tasks.rados.rados.0.trial039.stdout:275: finishing write tid 2 to trial03942481-36 2026-02-25T08:42:43.126 INFO:tasks.rados.rados.0.trial039.stdout:275: finishing write tid 3 to trial03942481-36 2026-02-25T08:42:43.128 INFO:tasks.rados.rados.0.trial039.stdout:275: finishing write tid 4 to trial03942481-36 2026-02-25T08:42:43.128 INFO:tasks.rados.rados.0.trial039.stdout:275: oid 36 updating version 0 to 45 2026-02-25T08:42:43.128 INFO:tasks.rados.rados.0.trial039.stdout:275: oid 36 version 45 is already newer than 43 2026-02-25T08:42:43.128 INFO:tasks.rados.rados.0.trial039.stdout:275: oid 36 version 45 is already newer than 45 2026-02-25T08:42:43.128 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 36 v 45 (ObjNum 150 snap 28 seq_num 150) dirty exists 2026-02-25T08:42:43.128 INFO:tasks.rados.rados.0.trial039.stdout:275: left oid 36 (ObjNum 150 snap 28 seq_num 150) 2026-02-25T08:42:44.052 INFO:tasks.rados.rados.0.trial039.stdout:273: done (2 left) 2026-02-25T08:42:44.053 INFO:tasks.rados.rados.0.trial039.stdout:275: done (1 left) 2026-02-25T08:42:44.053 INFO:tasks.rados.rados.0.trial039.stdout:276: done (0 left) 2026-02-25T08:42:44.053 INFO:tasks.rados.rados.0.trial039.stdout:277: read oid 26 snap 28 2026-02-25T08:42:44.053 INFO:tasks.rados.rados.0.trial039.stdout:277: expect (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:42:44.053 INFO:tasks.rados.rados.0.trial039.stdout:278: write (excl) oid 34 current snap is 29 2026-02-25T08:42:44.053 INFO:tasks.rados.rados.0.trial039.stdout:278: seq_num 151 ranges {537857=773306,1936428=520715} 2026-02-25T08:42:44.058 INFO:tasks.rados.rados.0.trial039.stdout:278: writing trial03942481-34 from 537857 to 1311163 tid 1 2026-02-25T08:42:44.063 INFO:tasks.rados.rados.0.trial039.stdout:278: writing trial03942481-34 from 1936428 to 2457143 tid 2 2026-02-25T08:42:44.063 INFO:tasks.rados.rados.0.trial039.stdout:279: write (excl) oid 5 current snap is 29 2026-02-25T08:42:44.063 INFO:tasks.rados.rados.0.trial039.stdout:279: seq_num 152 ranges {554664=448555,1478557=664373} 2026-02-25T08:42:44.066 INFO:tasks.rados.rados.0.trial039.stdout:279: writing trial03942481-5 from 554664 to 1003219 tid 1 2026-02-25T08:42:44.071 INFO:tasks.rados.rados.0.trial039.stdout:279: writing trial03942481-5 from 1478557 to 2142930 tid 2 2026-02-25T08:42:44.071 INFO:tasks.rados.rados.0.trial039.stdout:280: write oid 16 current snap is 29 2026-02-25T08:42:44.071 INFO:tasks.rados.rados.0.trial039.stdout:280: seq_num 153 ranges {571471=640157,1653392=524384} 2026-02-25T08:42:44.075 INFO:tasks.rados.rados.0.trial039.stdout:280: writing trial03942481-16 from 571471 to 1211628 tid 1 2026-02-25T08:42:44.079 INFO:tasks.rados.rados.0.trial039.stdout:280: writing trial03942481-16 from 1653392 to 2177776 tid 2 2026-02-25T08:42:44.079 INFO:tasks.rados.rados.0.trial039.stdout:281: snap_remove snap 27 2026-02-25T08:42:44.378 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:44.456 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:44.453+0000 7f386ddf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:44.458 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:44.457+0000 7f386ddf5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:44.484 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:44.501 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:44.829 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:44.901 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:44.903 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:44.904 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:45.056 INFO:tasks.rados.rados.0.trial039.stdout:281: done (4 left) 2026-02-25T08:42:45.057 INFO:tasks.rados.rados.0.trial039.stdout:282: snap_remove snap 25 2026-02-25T08:42:46.057 INFO:tasks.rados.rados.0.trial039.stdout:282: done (4 left) 2026-02-25T08:42:46.058 INFO:tasks.rados.rados.0.trial039.stdout:283: snap_remove snap 26 2026-02-25T08:42:46.502 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:46.580 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:46.578+0000 7f11d825a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:46.583 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:46.581+0000 7f11d825a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:46.609 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:46.625 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:46.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~72s 2026-02-25T08:42:47.065 INFO:tasks.rados.rados.0.trial039.stdout:283: done (4 left) 2026-02-25T08:42:47.065 INFO:tasks.rados.rados.0.trial039.stdout:284: write oid 33 current snap is 29 2026-02-25T08:42:47.065 INFO:tasks.rados.rados.0.trial039.stdout:279: finishing write tid 1 to trial03942481-5 2026-02-25T08:42:47.065 INFO:tasks.rados.rados.0.trial039.stdout:284: seq_num 154 ranges {588278=715406,1711874=668042} 2026-02-25T08:42:47.070 INFO:tasks.rados.rados.0.trial039.stdout:284: writing trial03942481-33 from 588278 to 1303684 tid 1 2026-02-25T08:42:47.074 INFO:tasks.rados.rados.0.trial039.stdout:284: writing trial03942481-33 from 1711874 to 2379916 tid 2 2026-02-25T08:42:47.075 INFO:tasks.rados.rados.0.trial039.stdout:285: write oid 23 current snap is 29 2026-02-25T08:42:47.075 INFO:tasks.rados.rados.0.trial039.stdout:285: seq_num 155 ranges {605085=790655,2170356=434729} 2026-02-25T08:42:47.080 INFO:tasks.rados.rados.0.trial039.stdout:285: writing trial03942481-23 from 605085 to 1395740 tid 1 2026-02-25T08:42:47.085 INFO:tasks.rados.rados.0.trial039.stdout:285: writing trial03942481-23 from 2170356 to 2605085 tid 2 2026-02-25T08:42:47.085 INFO:tasks.rados.rados.0.trial039.stdout:286: delete oid 21 current snap is 29 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:280: finishing write tid 1 to trial03942481-16 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:279: finishing write tid 2 to trial03942481-5 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:280: finishing write tid 2 to trial03942481-16 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:280: finishing write tid 3 to trial03942481-16 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:280: finishing write tid 4 to trial03942481-16 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:280: oid 16 updating version 0 to 23 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:280: oid 16 version 23 is already newer than 23 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:280: oid 16 version 23 is already newer than 23 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 16 v 23 (ObjNum 153 snap 29 seq_num 153) dirty exists 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:280: left oid 16 (ObjNum 153 snap 29 seq_num 153) 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:279: finishing write tid 3 to trial03942481-5 2026-02-25T08:42:47.086 INFO:tasks.rados.rados.0.trial039.stdout:279: finishing write tid 4 to trial03942481-5 2026-02-25T08:42:47.087 INFO:tasks.rados.rados.0.trial039.stdout:279: oid 5 updating version 0 to 28 2026-02-25T08:42:47.087 INFO:tasks.rados.rados.0.trial039.stdout:279: oid 5 version 28 is already newer than 26 2026-02-25T08:42:47.087 INFO:tasks.rados.rados.0.trial039.stdout:279: oid 5 version 28 is already newer than 28 2026-02-25T08:42:47.087 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 5 v 28 (ObjNum 152 snap 29 seq_num 152) dirty exists 2026-02-25T08:42:47.087 INFO:tasks.rados.rados.0.trial039.stdout:279: left oid 5 (ObjNum 152 snap 29 seq_num 152) 2026-02-25T08:42:47.087 INFO:tasks.rados.rados.0.trial039.stdout:277: expect (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:42:47.087 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:47.110 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:47.135 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:47.159 INFO:tasks.rados.rados.0.trial039.stdout:278: finishing write tid 1 to trial03942481-34 2026-02-25T08:42:47.159 INFO:tasks.rados.rados.0.trial039.stdout:278: finishing write tid 2 to trial03942481-34 2026-02-25T08:42:47.159 INFO:tasks.rados.rados.0.trial039.stdout:277: done (6 left) 2026-02-25T08:42:47.159 INFO:tasks.rados.rados.0.trial039.stdout:279: done (5 left) 2026-02-25T08:42:47.160 INFO:tasks.rados.rados.0.trial039.stdout:280: done (4 left) 2026-02-25T08:42:47.160 INFO:tasks.rados.rados.0.trial039.stdout:286: done (3 left) 2026-02-25T08:42:47.160 INFO:tasks.rados.rados.0.trial039.stdout:287: snap_remove snap 28 2026-02-25T08:42:48.065 INFO:tasks.rados.rados.0.trial039.stdout:287: done (3 left) 2026-02-25T08:42:48.066 INFO:tasks.rados.rados.0.trial039.stdout:288: read oid 43 snap -1 2026-02-25T08:42:48.066 INFO:tasks.rados.rados.0.trial039.stdout:288: expect deleted 2026-02-25T08:42:48.066 INFO:tasks.rados.rados.0.trial039.stdout:289: write oid 13 current snap is 29 2026-02-25T08:42:48.066 INFO:tasks.rados.rados.0.trial039.stdout:278: finishing write tid 3 to trial03942481-34 2026-02-25T08:42:48.066 INFO:tasks.rados.rados.0.trial039.stdout:289: seq_num 157 ranges {638699=541153,1770967=531722} 2026-02-25T08:42:48.069 INFO:tasks.rados.rados.0.trial039.stdout:289: writing trial03942481-13 from 638699 to 1179852 tid 1 2026-02-25T08:42:48.074 INFO:tasks.rados.rados.0.trial039.stdout:289: writing trial03942481-13 from 1770967 to 2302689 tid 2 2026-02-25T08:42:48.074 INFO:tasks.rados.rados.0.trial039.stdout:290: write (excl) oid 18 current snap is 29 2026-02-25T08:42:48.074 INFO:tasks.rados.rados.0.trial039.stdout:278: finishing write tid 4 to trial03942481-34 2026-02-25T08:42:48.075 INFO:tasks.rados.rados.0.trial039.stdout:278: oid 34 updating version 0 to 67 2026-02-25T08:42:48.075 INFO:tasks.rados.rados.0.trial039.stdout:278: oid 34 version 67 is already newer than 65 2026-02-25T08:42:48.075 INFO:tasks.rados.rados.0.trial039.stdout:278: oid 34 version 67 is already newer than 67 2026-02-25T08:42:48.075 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 67 (ObjNum 151 snap 29 seq_num 151) dirty exists 2026-02-25T08:42:48.075 INFO:tasks.rados.rados.0.trial039.stdout:278: left oid 34 (ObjNum 151 snap 29 seq_num 151) 2026-02-25T08:42:48.075 INFO:tasks.rados.rados.0.trial039.stdout:290: seq_num 158 ranges {655506=616402,1829449=675380} 2026-02-25T08:42:48.079 INFO:tasks.rados.rados.0.trial039.stdout:290: writing trial03942481-18 from 655506 to 1271908 tid 1 2026-02-25T08:42:48.084 INFO:tasks.rados.rados.0.trial039.stdout:290: writing trial03942481-18 from 1829449 to 2504829 tid 2 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:278: done (5 left) 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:291: read oid 26 snap -1 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:291: expect (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:292: snap_create 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:284: finishing write tid 1 to trial03942481-33 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:284: finishing write tid 2 to trial03942481-33 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:284: finishing write tid 3 to trial03942481-33 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:284: finishing write tid 4 to trial03942481-33 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:284: oid 33 updating version 0 to 40 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:284: oid 33 version 40 is already newer than 40 2026-02-25T08:42:48.085 INFO:tasks.rados.rados.0.trial039.stdout:284: oid 33 version 40 is already newer than 40 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 33 v 40 (ObjNum 154 snap 29 seq_num 154) dirty exists 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:284: left oid 33 (ObjNum 154 snap 29 seq_num 154) 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:285: finishing write tid 1 to trial03942481-23 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:285: finishing write tid 2 to trial03942481-23 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:285: finishing write tid 3 to trial03942481-23 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:285: finishing write tid 4 to trial03942481-23 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:285: oid 23 updating version 0 to 32 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:285: oid 23 version 32 is already newer than 32 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:285: oid 23 version 32 is already newer than 32 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 23 v 32 (ObjNum 155 snap 29 seq_num 155) dirty exists 2026-02-25T08:42:48.086 INFO:tasks.rados.rados.0.trial039.stdout:285: left oid 23 (ObjNum 155 snap 29 seq_num 155) 2026-02-25T08:42:48.104 INFO:tasks.rados.rados.0.trial039.stdout:291: expect (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:42:48.107 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 632548 2026-02-25T08:42:48.131 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 856296 2026-02-25T08:42:48.145 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:48.170 INFO:tasks.rados.rados.0.trial039.stdout:289: finishing write tid 1 to trial03942481-13 2026-02-25T08:42:48.170 INFO:tasks.rados.rados.0.trial039.stdout:289: finishing write tid 2 to trial03942481-13 2026-02-25T08:42:48.170 INFO:tasks.rados.rados.0.trial039.stdout:289: finishing write tid 3 to trial03942481-13 2026-02-25T08:42:48.170 INFO:tasks.rados.rados.0.trial039.stdout:289: finishing write tid 4 to trial03942481-13 2026-02-25T08:42:48.170 INFO:tasks.rados.rados.0.trial039.stdout:289: oid 13 updating version 0 to 48 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:289: oid 13 version 48 is already newer than 48 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:289: oid 13 version 48 is already newer than 48 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 13 v 48 (ObjNum 157 snap 29 seq_num 157) dirty exists 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:289: left oid 13 (ObjNum 157 snap 29 seq_num 157) 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:290: finishing write tid 1 to trial03942481-18 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:290: finishing write tid 2 to trial03942481-18 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:290: finishing write tid 3 to trial03942481-18 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:290: finishing write tid 4 to trial03942481-18 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:290: oid 18 updating version 0 to 51 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:290: oid 18 version 51 is already newer than 49 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:290: oid 18 version 51 is already newer than 51 2026-02-25T08:42:48.171 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 18 v 51 (ObjNum 158 snap 29 seq_num 158) dirty exists 2026-02-25T08:42:48.172 INFO:tasks.rados.rados.0.trial039.stdout:290: left oid 18 (ObjNum 158 snap 29 seq_num 158) 2026-02-25T08:42:48.626 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:48.703 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:48.700+0000 7f6b9b37e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:48.705 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:48.704+0000 7f6b9b37e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:48.731 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:48.749 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:48.749 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:49.071 INFO:tasks.rados.rados.0.trial039.stdout:284: done (6 left) 2026-02-25T08:42:49.071 INFO:tasks.rados.rados.0.trial039.stdout:285: done (5 left) 2026-02-25T08:42:49.071 INFO:tasks.rados.rados.0.trial039.stdout:288: done (4 left) 2026-02-25T08:42:49.072 INFO:tasks.rados.rados.0.trial039.stdout:289: done (3 left) 2026-02-25T08:42:49.072 INFO:tasks.rados.rados.0.trial039.stdout:290: done (2 left) 2026-02-25T08:42:49.072 INFO:tasks.rados.rados.0.trial039.stdout:291: done (1 left) 2026-02-25T08:42:49.072 INFO:tasks.rados.rados.0.trial039.stdout:292: done (0 left) 2026-02-25T08:42:49.072 INFO:tasks.rados.rados.0.trial039.stdout:293: snap_remove snap 29 2026-02-25T08:42:49.905 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:49.978 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:49.981 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:50.074 INFO:tasks.rados.rados.0.trial039.stdout:293: done (0 left) 2026-02-25T08:42:50.074 INFO:tasks.rados.rados.0.trial039.stdout:294: delete oid 9 current snap is 30 2026-02-25T08:42:50.105 INFO:tasks.rados.rados.0.trial039.stdout:294: done (0 left) 2026-02-25T08:42:50.105 INFO:tasks.rados.rados.0.trial039.stdout:295: write oid 13 current snap is 30 2026-02-25T08:42:50.105 INFO:tasks.rados.rados.0.trial039.stdout:295: seq_num 160 ranges {689120=483253,1946413=679049} 2026-02-25T08:42:50.110 INFO:tasks.rados.rados.0.trial039.stdout:295: writing trial03942481-13 from 689120 to 1172373 tid 1 2026-02-25T08:42:50.115 INFO:tasks.rados.rados.0.trial039.stdout:295: writing trial03942481-13 from 1946413 to 2625462 tid 2 2026-02-25T08:42:50.115 INFO:tasks.rados.rados.0.trial039.stdout:296: write (excl) oid 14 current snap is 30 2026-02-25T08:42:50.116 INFO:tasks.rados.rados.0.trial039.stdout:296: seq_num 161 ranges {705927=558502,2004895=422707} 2026-02-25T08:42:50.120 INFO:tasks.rados.rados.0.trial039.stdout:296: writing trial03942481-14 from 705927 to 1264429 tid 1 2026-02-25T08:42:50.125 INFO:tasks.rados.rados.0.trial039.stdout:296: writing trial03942481-14 from 2004895 to 2427602 tid 2 2026-02-25T08:42:50.126 INFO:tasks.rados.rados.0.trial039.stdout:297: read oid 27 snap -1 2026-02-25T08:42:50.126 INFO:tasks.rados.rados.0.trial039.stdout:297: expect (ObjNum 149 snap 27 seq_num 149) 2026-02-25T08:42:50.126 INFO:tasks.rados.rados.0.trial039.stdout:298: snap_create 2026-02-25T08:42:50.134 INFO:tasks.rados.rados.0.trial039.stdout:295: finishing write tid 1 to trial03942481-13 2026-02-25T08:42:50.141 INFO:tasks.rados.rados.0.trial039.stdout:295: finishing write tid 2 to trial03942481-13 2026-02-25T08:42:50.144 INFO:tasks.rados.rados.0.trial039.stdout:296: finishing write tid 1 to trial03942481-14 2026-02-25T08:42:50.151 INFO:tasks.rados.rados.0.trial039.stdout:297: expect (ObjNum 149 snap 27 seq_num 149) 2026-02-25T08:42:50.173 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:42:50.215 INFO:tasks.rados.rados.0.trial039.stdout:295: finishing write tid 3 to trial03942481-13 2026-02-25T08:42:50.215 INFO:tasks.rados.rados.0.trial039.stdout:295: finishing write tid 4 to trial03942481-13 2026-02-25T08:42:50.215 INFO:tasks.rados.rados.0.trial039.stdout:295: oid 13 updating version 0 to 54 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:295: oid 13 version 54 is already newer than 54 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:295: oid 13 version 54 is already newer than 54 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 13 v 54 (ObjNum 160 snap 30 seq_num 160) dirty exists 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:295: left oid 13 (ObjNum 160 snap 30 seq_num 160) 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:296: finishing write tid 2 to trial03942481-14 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:296: finishing write tid 3 to trial03942481-14 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:296: finishing write tid 4 to trial03942481-14 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:296: oid 14 updating version 0 to 70 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:296: oid 14 version 70 is already newer than 68 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:296: oid 14 version 70 is already newer than 70 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 14 v 70 (ObjNum 161 snap 30 seq_num 161) dirty exists 2026-02-25T08:42:50.216 INFO:tasks.rados.rados.0.trial039.stdout:296: left oid 14 (ObjNum 161 snap 30 seq_num 161) 2026-02-25T08:42:50.751 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:50.827 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:50.825+0000 7f5c0650b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:50.830 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:50.829+0000 7f5c0650b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:50.857 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:50.874 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:51.080 INFO:tasks.rados.rados.0.trial039.stdout:295: done (3 left) 2026-02-25T08:42:51.080 INFO:tasks.rados.rados.0.trial039.stdout:296: done (2 left) 2026-02-25T08:42:51.080 INFO:tasks.rados.rados.0.trial039.stdout:297: done (1 left) 2026-02-25T08:42:51.081 INFO:tasks.rados.rados.0.trial039.stdout:298: done (0 left) 2026-02-25T08:42:51.081 INFO:tasks.rados.rados.0.trial039.stdout:299: snap_remove snap 30 2026-02-25T08:42:52.083 INFO:tasks.rados.rados.0.trial039.stdout:299: done (0 left) 2026-02-25T08:42:52.083 INFO:tasks.rados.rados.0.trial039.stdout:300: read oid 22 snap -1 2026-02-25T08:42:52.083 INFO:tasks.rados.rados.0.trial039.stdout:300: expect (ObjNum 145 snap 27 seq_num 145) 2026-02-25T08:42:52.083 INFO:tasks.rados.rados.0.trial039.stdout:301: write oid 10 current snap is 31 2026-02-25T08:42:52.083 INFO:tasks.rados.rados.0.trial039.stdout:301: seq_num 162 ranges {722734=633751,2063377=659357} 2026-02-25T08:42:52.087 INFO:tasks.rados.rados.0.trial039.stdout:301: writing trial03942481-10 from 722734 to 1356485 tid 1 2026-02-25T08:42:52.093 INFO:tasks.rados.rados.0.trial039.stdout:301: writing trial03942481-10 from 2063377 to 2722734 tid 2 2026-02-25T08:42:52.093 INFO:tasks.rados.rados.0.trial039.stdout:302: delete oid 15 current snap is 31 2026-02-25T08:42:52.119 INFO:tasks.rados.rados.0.trial039.stdout:302: done (2 left) 2026-02-25T08:42:52.119 INFO:tasks.rados.rados.0.trial039.stdout:303: snap_create 2026-02-25T08:42:52.119 INFO:tasks.rados.rados.0.trial039.stdout:300: expect (ObjNum 145 snap 27 seq_num 145) 2026-02-25T08:42:52.131 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1803421 2026-02-25T08:42:52.151 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1812836 2026-02-25T08:42:52.166 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1383084 2026-02-25T08:42:52.184 INFO:tasks.rados.rados.0.trial039.stdout:301: finishing write tid 1 to trial03942481-10 2026-02-25T08:42:52.184 INFO:tasks.rados.rados.0.trial039.stdout:301: finishing write tid 2 to trial03942481-10 2026-02-25T08:42:52.184 INFO:tasks.rados.rados.0.trial039.stdout:301: finishing write tid 3 to trial03942481-10 2026-02-25T08:42:52.184 INFO:tasks.rados.rados.0.trial039.stdout:301: finishing write tid 4 to trial03942481-10 2026-02-25T08:42:52.184 INFO:tasks.rados.rados.0.trial039.stdout:301: oid 10 updating version 0 to 43 2026-02-25T08:42:52.184 INFO:tasks.rados.rados.0.trial039.stdout:301: oid 10 version 43 is already newer than 43 2026-02-25T08:42:52.184 INFO:tasks.rados.rados.0.trial039.stdout:301: oid 10 version 43 is already newer than 43 2026-02-25T08:42:52.184 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 10 v 43 (ObjNum 162 snap 31 seq_num 162) dirty exists 2026-02-25T08:42:52.184 INFO:tasks.rados.rados.0.trial039.stdout:301: left oid 10 (ObjNum 162 snap 31 seq_num 162) 2026-02-25T08:42:52.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~77s 2026-02-25T08:42:52.875 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:52.953 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:52.950+0000 7f5cf69dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:52.955 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:52.954+0000 7f5cf5753640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:52.981 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:52.998 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:52.998 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:53.053 INFO:tasks.rados.rados.0.trial039.stdout:300: done (2 left) 2026-02-25T08:42:53.053 INFO:tasks.rados.rados.0.trial039.stdout:301: done (1 left) 2026-02-25T08:42:53.053 INFO:tasks.rados.rados.0.trial039.stdout:303: done (0 left) 2026-02-25T08:42:53.053 INFO:tasks.rados.rados.0.trial039.stdout:304: write (excl) oid 13 current snap is 32 2026-02-25T08:42:53.054 INFO:tasks.rados.rados.0.trial039.stdout:304: seq_num 164 ranges {756348=784249,2063988=686387} 2026-02-25T08:42:53.059 INFO:tasks.rados.rados.0.trial039.stdout:304: writing trial03942481-13 from 756348 to 1540597 tid 1 2026-02-25T08:42:53.065 INFO:tasks.rados.rados.0.trial039.stdout:304: writing trial03942481-13 from 2063988 to 2750375 tid 2 2026-02-25T08:42:53.065 INFO:tasks.rados.rados.0.trial039.stdout:305: write oid 48 current snap is 32 2026-02-25T08:42:53.065 INFO:tasks.rados.rados.0.trial039.stdout:305: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2026-02-25T08:42:53.069 INFO:tasks.rados.rados.0.trial039.stdout:305: writing trial03942481-48 from 773155 to 1232653 tid 1 2026-02-25T08:42:53.073 INFO:tasks.rados.rados.0.trial039.stdout:305: writing trial03942481-48 from 1722470 to 2152515 tid 2 2026-02-25T08:42:53.075 INFO:tasks.rados.rados.0.trial039.stdout:305: writing trial03942481-48 from 2648676 to 2773155 tid 3 2026-02-25T08:42:53.076 INFO:tasks.rados.rados.0.trial039.stdout:306: write (excl) oid 12 current snap is 32 2026-02-25T08:42:53.076 INFO:tasks.rados.rados.0.trial039.stdout:306: seq_num 166 ranges {789962=534747,1780952=690056} 2026-02-25T08:42:53.080 INFO:tasks.rados.rados.0.trial039.stdout:306: writing trial03942481-12 from 789962 to 1324709 tid 1 2026-02-25T08:42:53.085 INFO:tasks.rados.rados.0.trial039.stdout:306: writing trial03942481-12 from 1780952 to 2471008 tid 2 2026-02-25T08:42:53.085 INFO:tasks.rados.rados.0.trial039.stdout:307: snap_remove snap 31 2026-02-25T08:42:54.092 INFO:tasks.rados.rados.0.trial039.stdout:307: done (3 left) 2026-02-25T08:42:54.092 INFO:tasks.rados.rados.0.trial039.stdout:308: snap_create 2026-02-25T08:42:54.092 INFO:tasks.rados.rados.0.trial039.stdout:305: finishing write tid 1 to trial03942481-48 2026-02-25T08:42:54.092 INFO:tasks.rados.rados.0.trial039.stdout:305: finishing write tid 2 to trial03942481-48 2026-02-25T08:42:54.092 INFO:tasks.rados.rados.0.trial039.stdout:305: finishing write tid 3 to trial03942481-48 2026-02-25T08:42:54.092 INFO:tasks.rados.rados.0.trial039.stdout:305: finishing write tid 4 to trial03942481-48 2026-02-25T08:42:54.092 INFO:tasks.rados.rados.0.trial039.stdout:305: finishing write tid 5 to trial03942481-48 2026-02-25T08:42:54.092 INFO:tasks.rados.rados.0.trial039.stdout:305: oid 48 updating version 0 to 16 2026-02-25T08:42:54.092 INFO:tasks.rados.rados.0.trial039.stdout:305: oid 48 version 16 is already newer than 16 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:305: oid 48 version 16 is already newer than 16 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:305: oid 48 version 16 is already newer than 16 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 48 v 16 (ObjNum 165 snap 32 seq_num 165) dirty exists 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:305: left oid 48 (ObjNum 165 snap 32 seq_num 165) 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:304: finishing write tid 1 to trial03942481-13 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:304: finishing write tid 2 to trial03942481-13 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:304: finishing write tid 3 to trial03942481-13 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:304: finishing write tid 4 to trial03942481-13 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:304: oid 13 updating version 0 to 59 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:304: oid 13 version 59 is already newer than 57 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:304: oid 13 version 59 is already newer than 59 2026-02-25T08:42:54.093 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 13 v 59 (ObjNum 164 snap 32 seq_num 164) dirty exists 2026-02-25T08:42:54.094 INFO:tasks.rados.rados.0.trial039.stdout:304: left oid 13 (ObjNum 164 snap 32 seq_num 164) 2026-02-25T08:42:54.094 INFO:tasks.rados.rados.0.trial039.stdout:306: finishing write tid 1 to trial03942481-12 2026-02-25T08:42:54.094 INFO:tasks.rados.rados.0.trial039.stdout:306: finishing write tid 2 to trial03942481-12 2026-02-25T08:42:54.094 INFO:tasks.rados.rados.0.trial039.stdout:306: finishing write tid 3 to trial03942481-12 2026-02-25T08:42:54.094 INFO:tasks.rados.rados.0.trial039.stdout:306: finishing write tid 4 to trial03942481-12 2026-02-25T08:42:54.094 INFO:tasks.rados.rados.0.trial039.stdout:306: oid 12 updating version 0 to 27 2026-02-25T08:42:54.094 INFO:tasks.rados.rados.0.trial039.stdout:306: oid 12 version 27 is already newer than 25 2026-02-25T08:42:54.094 INFO:tasks.rados.rados.0.trial039.stdout:306: oid 12 version 27 is already newer than 27 2026-02-25T08:42:54.094 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 12 v 27 (ObjNum 166 snap 32 seq_num 166) dirty exists 2026-02-25T08:42:54.094 INFO:tasks.rados.rados.0.trial039.stdout:306: left oid 12 (ObjNum 166 snap 32 seq_num 166) 2026-02-25T08:42:54.982 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:42:54.999 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:55.055 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:55.058 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:42:55.077 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:55.073+0000 7f0cb32cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:55.079 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:55.077+0000 7f0cb32cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:55.092 INFO:tasks.rados.rados.0.trial039.stdout:304: done (3 left) 2026-02-25T08:42:55.092 INFO:tasks.rados.rados.0.trial039.stdout:305: done (2 left) 2026-02-25T08:42:55.092 INFO:tasks.rados.rados.0.trial039.stdout:306: done (1 left) 2026-02-25T08:42:55.092 INFO:tasks.rados.rados.0.trial039.stdout:308: done (0 left) 2026-02-25T08:42:55.092 INFO:tasks.rados.rados.0.trial039.stdout:309: snap_create 2026-02-25T08:42:55.117 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:55.133 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:56.099 INFO:tasks.rados.rados.0.trial039.stdout:309: done (0 left) 2026-02-25T08:42:56.100 INFO:tasks.rados.rados.0.trial039.stdout:310: write (excl) oid 14 current snap is 34 2026-02-25T08:42:56.100 INFO:tasks.rados.rados.0.trial039.stdout:310: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2026-02-25T08:42:56.103 INFO:tasks.rados.rados.0.trial039.stdout:310: writing trial03942481-14 from 406769 to 1016765 tid 1 2026-02-25T08:42:56.107 INFO:tasks.rados.rados.0.trial039.stdout:310: writing trial03942481-14 from 1439434 to 1873148 tid 2 2026-02-25T08:42:56.112 INFO:tasks.rados.rados.0.trial039.stdout:310: writing trial03942481-14 from 2303522 to 2806769 tid 3 2026-02-25T08:42:56.112 INFO:tasks.rados.rados.0.trial039.stdout:311: write (excl) oid 17 current snap is 34 2026-02-25T08:42:56.112 INFO:tasks.rados.rados.0.trial039.stdout:311: seq_num 168 ranges {423576=401598,1497916=693725} 2026-02-25T08:42:56.114 INFO:tasks.rados.rados.0.trial039.stdout:311: writing trial03942481-17 from 423576 to 825174 tid 1 2026-02-25T08:42:56.120 INFO:tasks.rados.rados.0.trial039.stdout:311: writing trial03942481-17 from 1497916 to 2191641 tid 2 2026-02-25T08:42:56.120 INFO:tasks.rados.rados.0.trial039.stdout:312: write oid 24 current snap is 34 2026-02-25T08:42:56.120 INFO:tasks.rados.rados.0.trial039.stdout:312: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2026-02-25T08:42:56.123 INFO:tasks.rados.rados.0.trial039.stdout:312: writing trial03942481-24 from 440383 to 917230 tid 1 2026-02-25T08:42:56.127 INFO:tasks.rados.rados.0.trial039.stdout:312: writing trial03942481-24 from 1556398 to 1993781 tid 2 2026-02-25T08:42:56.131 INFO:tasks.rados.rados.0.trial039.stdout:312: writing trial03942481-24 from 2474721 to 2840383 tid 3 2026-02-25T08:42:56.131 INFO:tasks.rados.rados.0.trial039.stdout:313: snap_remove snap 33 2026-02-25T08:42:57.105 INFO:tasks.rados.rados.0.trial039.stdout:313: done (3 left) 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:314: snap_create 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:311: finishing write tid 1 to trial03942481-17 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:311: finishing write tid 2 to trial03942481-17 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:311: finishing write tid 3 to trial03942481-17 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:310: finishing write tid 1 to trial03942481-14 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:311: finishing write tid 4 to trial03942481-17 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:311: oid 17 updating version 0 to 60 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:311: oid 17 version 60 is already newer than 58 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:311: oid 17 version 60 is already newer than 60 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 17 v 60 (ObjNum 168 snap 34 seq_num 168) dirty exists 2026-02-25T08:42:57.106 INFO:tasks.rados.rados.0.trial039.stdout:311: left oid 17 (ObjNum 168 snap 34 seq_num 168) 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:310: finishing write tid 2 to trial03942481-14 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:310: finishing write tid 3 to trial03942481-14 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:310: finishing write tid 4 to trial03942481-14 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:310: finishing write tid 5 to trial03942481-14 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:310: oid 14 updating version 0 to 75 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:310: oid 14 version 75 is already newer than 75 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:310: oid 14 version 75 is already newer than 75 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:310: oid 14 version 75 is already newer than 72 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 14 v 75 (ObjNum 167 snap 34 seq_num 167) dirty exists 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:310: left oid 14 (ObjNum 167 snap 34 seq_num 167) 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:312: finishing write tid 1 to trial03942481-24 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:312: finishing write tid 2 to trial03942481-24 2026-02-25T08:42:57.107 INFO:tasks.rados.rados.0.trial039.stdout:312: finishing write tid 3 to trial03942481-24 2026-02-25T08:42:57.108 INFO:tasks.rados.rados.0.trial039.stdout:312: finishing write tid 4 to trial03942481-24 2026-02-25T08:42:57.108 INFO:tasks.rados.rados.0.trial039.stdout:312: finishing write tid 5 to trial03942481-24 2026-02-25T08:42:57.108 INFO:tasks.rados.rados.0.trial039.stdout:312: oid 24 updating version 0 to 34 2026-02-25T08:42:57.108 INFO:tasks.rados.rados.0.trial039.stdout:312: oid 24 version 34 is already newer than 34 2026-02-25T08:42:57.108 INFO:tasks.rados.rados.0.trial039.stdout:312: oid 24 version 34 is already newer than 34 2026-02-25T08:42:57.108 INFO:tasks.rados.rados.0.trial039.stdout:312: oid 24 version 34 is already newer than 34 2026-02-25T08:42:57.108 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 24 v 34 (ObjNum 169 snap 34 seq_num 169) dirty exists 2026-02-25T08:42:57.108 INFO:tasks.rados.rados.0.trial039.stdout:312: left oid 24 (ObjNum 169 snap 34 seq_num 169) 2026-02-25T08:42:57.135 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:57.209 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:57.205+0000 7f13331dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:57.211 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:57.209+0000 7f13331dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:57.238 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:57.255 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:57.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~83s 2026-02-25T08:42:58.104 INFO:tasks.rados.rados.0.trial039.stdout:310: done (3 left) 2026-02-25T08:42:58.104 INFO:tasks.rados.rados.0.trial039.stdout:311: done (2 left) 2026-02-25T08:42:58.104 INFO:tasks.rados.rados.0.trial039.stdout:312: done (1 left) 2026-02-25T08:42:58.104 INFO:tasks.rados.rados.0.trial039.stdout:314: done (0 left) 2026-02-25T08:42:58.104 INFO:tasks.rados.rados.0.trial039.stdout:315: delete oid 12 current snap is 35 2026-02-25T08:42:58.144 INFO:tasks.rados.rados.0.trial039.stdout:315: done (0 left) 2026-02-25T08:42:58.144 INFO:tasks.rados.rados.0.trial039.stdout:316: write (excl) oid 43 current snap is 35 2026-02-25T08:42:58.144 INFO:tasks.rados.rados.0.trial039.stdout:316: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2026-02-25T08:42:58.148 INFO:tasks.rados.rados.0.trial039.stdout:316: writing trial03942481-43 from 473997 to 1101342 tid 1 2026-02-25T08:42:58.152 INFO:tasks.rados.rados.0.trial039.stdout:316: writing trial03942481-43 from 1673362 to 2114414 tid 2 2026-02-25T08:42:58.157 INFO:tasks.rados.rados.0.trial039.stdout:316: writing trial03942481-43 from 2529567 to 2873997 tid 3 2026-02-25T08:42:58.157 INFO:tasks.rados.rados.0.trial039.stdout:317: snap_create 2026-02-25T08:42:58.163 INFO:tasks.rados.rados.0.trial039.stdout:316: finishing write tid 1 to trial03942481-43 2026-02-25T08:42:58.176 INFO:tasks.rados.rados.0.trial039.stdout:316: finishing write tid 2 to trial03942481-43 2026-02-25T08:42:58.181 INFO:tasks.rados.rados.0.trial039.stdout:316: finishing write tid 3 to trial03942481-43 2026-02-25T08:42:58.186 INFO:tasks.rados.rados.0.trial039.stdout:316: finishing write tid 4 to trial03942481-43 2026-02-25T08:42:58.188 INFO:tasks.rados.rados.0.trial039.stdout:316: finishing write tid 5 to trial03942481-43 2026-02-25T08:42:58.188 INFO:tasks.rados.rados.0.trial039.stdout:316: oid 43 updating version 0 to 47 2026-02-25T08:42:58.188 INFO:tasks.rados.rados.0.trial039.stdout:316: oid 43 version 47 is already newer than 44 2026-02-25T08:42:58.188 INFO:tasks.rados.rados.0.trial039.stdout:316: oid 43 version 47 is already newer than 47 2026-02-25T08:42:58.188 INFO:tasks.rados.rados.0.trial039.stdout:316: oid 43 version 47 is already newer than 47 2026-02-25T08:42:58.188 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 43 v 47 (ObjNum 171 snap 35 seq_num 171) dirty exists 2026-02-25T08:42:58.188 INFO:tasks.rados.rados.0.trial039.stdout:316: left oid 43 (ObjNum 171 snap 35 seq_num 171) 2026-02-25T08:42:59.110 INFO:tasks.rados.rados.0.trial039.stdout:316: done (1 left) 2026-02-25T08:42:59.110 INFO:tasks.rados.rados.0.trial039.stdout:317: done (0 left) 2026-02-25T08:42:59.110 INFO:tasks.rados.rados.0.trial039.stdout:318: snap_create 2026-02-25T08:42:59.256 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:59.331 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:59.328+0000 7fab51ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:59.334 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:42:59.331+0000 7fab51ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:59.360 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:42:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:59.376 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:00.060 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:00.113 INFO:tasks.rados.rados.0.trial039.stdout:318: done (0 left) 2026-02-25T08:43:00.113 INFO:tasks.rados.rados.0.trial039.stdout:319: write oid 45 current snap is 37 2026-02-25T08:43:00.113 INFO:tasks.rados.rados.0.trial039.stdout:319: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2026-02-25T08:43:00.118 INFO:tasks.rados.rados.0.trial039.stdout:319: writing trial03942481-45 from 490804 to 1193398 tid 1 2026-02-25T08:43:00.123 INFO:tasks.rados.rados.0.trial039.stdout:319: writing trial03942481-45 from 1615491 to 2200201 tid 2 2026-02-25T08:43:00.126 INFO:tasks.rados.rados.0.trial039.stdout:319: writing trial03942481-45 from 2724284 to 2890804 tid 3 2026-02-25T08:43:00.126 INFO:tasks.rados.rados.0.trial039.stdout:320: snap_create 2026-02-25T08:43:00.135 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:00.138 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:00.169 INFO:tasks.rados.rados.0.trial039.stdout:319: finishing write tid 1 to trial03942481-45 2026-02-25T08:43:00.176 INFO:tasks.rados.rados.0.trial039.stdout:319: finishing write tid 2 to trial03942481-45 2026-02-25T08:43:00.183 INFO:tasks.rados.rados.0.trial039.stdout:319: finishing write tid 3 to trial03942481-45 2026-02-25T08:43:00.188 INFO:tasks.rados.rados.0.trial039.stdout:319: finishing write tid 4 to trial03942481-45 2026-02-25T08:43:00.190 INFO:tasks.rados.rados.0.trial039.stdout:319: finishing write tid 5 to trial03942481-45 2026-02-25T08:43:00.190 INFO:tasks.rados.rados.0.trial039.stdout:319: oid 45 updating version 0 to 80 2026-02-25T08:43:00.190 INFO:tasks.rados.rados.0.trial039.stdout:319: oid 45 version 80 is already newer than 80 2026-02-25T08:43:00.190 INFO:tasks.rados.rados.0.trial039.stdout:319: oid 45 version 80 is already newer than 80 2026-02-25T08:43:00.190 INFO:tasks.rados.rados.0.trial039.stdout:319: oid 45 version 80 is already newer than 80 2026-02-25T08:43:00.190 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 45 v 80 (ObjNum 172 snap 37 seq_num 172) dirty exists 2026-02-25T08:43:00.190 INFO:tasks.rados.rados.0.trial039.stdout:319: left oid 45 (ObjNum 172 snap 37 seq_num 172) 2026-02-25T08:43:01.118 INFO:tasks.rados.rados.0.trial039.stdout:319: done (1 left) 2026-02-25T08:43:01.119 INFO:tasks.rados.rados.0.trial039.stdout:320: done (0 left) 2026-02-25T08:43:01.119 INFO:tasks.rados.rados.0.trial039.stdout:321: snap_create 2026-02-25T08:43:01.377 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:01.453 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:01.450+0000 7fa56cbb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:01.456 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:01.454+0000 7fa56cbb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:01.482 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:01.499 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:02.124 INFO:tasks.rados.rados.0.trial039.stdout:321: done (0 left) 2026-02-25T08:43:02.124 INFO:tasks.rados.rados.0.trial039.stdout:322: delete oid 13 current snap is 39 2026-02-25T08:43:02.166 INFO:tasks.rados.rados.0.trial039.stdout:322: done (0 left) 2026-02-25T08:43:02.166 INFO:tasks.rados.rados.0.trial039.stdout:323: write (excl) oid 36 current snap is 39 2026-02-25T08:43:02.166 INFO:tasks.rados.rados.0.trial039.stdout:323: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2026-02-25T08:43:02.170 INFO:tasks.rados.rados.0.trial039.stdout:323: writing trial03942481-36 from 524418 to 977510 tid 1 2026-02-25T08:43:02.175 INFO:tasks.rados.rados.0.trial039.stdout:323: writing trial03942481-36 from 1732455 to 2320834 tid 2 2026-02-25T08:43:02.178 INFO:tasks.rados.rados.0.trial039.stdout:323: writing trial03942481-36 from 2779130 to 2924418 tid 3 2026-02-25T08:43:02.178 INFO:tasks.rados.rados.0.trial039.stdout:324: snap_create 2026-02-25T08:43:02.198 INFO:tasks.rados.rados.0.trial039.stdout:323: finishing write tid 1 to trial03942481-36 2026-02-25T08:43:02.217 INFO:tasks.rados.rados.0.trial039.stdout:323: finishing write tid 2 to trial03942481-36 2026-02-25T08:43:02.226 INFO:tasks.rados.rados.0.trial039.stdout:323: finishing write tid 3 to trial03942481-36 2026-02-25T08:43:02.231 INFO:tasks.rados.rados.0.trial039.stdout:323: finishing write tid 4 to trial03942481-36 2026-02-25T08:43:02.233 INFO:tasks.rados.rados.0.trial039.stdout:323: finishing write tid 5 to trial03942481-36 2026-02-25T08:43:02.233 INFO:tasks.rados.rados.0.trial039.stdout:323: oid 36 updating version 0 to 50 2026-02-25T08:43:02.233 INFO:tasks.rados.rados.0.trial039.stdout:323: oid 36 version 50 is already newer than 50 2026-02-25T08:43:02.233 INFO:tasks.rados.rados.0.trial039.stdout:323: oid 36 version 50 is already newer than 50 2026-02-25T08:43:02.233 INFO:tasks.rados.rados.0.trial039.stdout:323: oid 36 version 50 is already newer than 47 2026-02-25T08:43:02.233 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 36 v 50 (ObjNum 174 snap 39 seq_num 174) dirty exists 2026-02-25T08:43:02.233 INFO:tasks.rados.rados.0.trial039.stdout:323: left oid 36 (ObjNum 174 snap 39 seq_num 174) 2026-02-25T08:43:03.056 INFO:tasks.rados.rados.0.trial039.stdout:323: done (1 left) 2026-02-25T08:43:03.057 INFO:tasks.rados.rados.0.trial039.stdout:324: done (0 left) 2026-02-25T08:43:03.057 INFO:tasks.rados.rados.0.trial039.stdout:325: read oid 48 snap -1 2026-02-25T08:43:03.057 INFO:tasks.rados.rados.0.trial039.stdout:325: expect (ObjNum 165 snap 32 seq_num 165) 2026-02-25T08:43:03.057 INFO:tasks.rados.rados.0.trial039.stdout:326: snap_remove snap 36 2026-02-25T08:43:03.398 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~88s 2026-02-25T08:43:03.499 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:03.577 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:03.574+0000 7f164450b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:03.580 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:03.577+0000 7f164450b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:03.605 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:03.623 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:04.057 INFO:tasks.rados.rados.0.trial039.stdout:326: done (1 left) 2026-02-25T08:43:04.057 INFO:tasks.rados.rados.0.trial039.stdout:327: write oid 27 current snap is 40 2026-02-25T08:43:04.058 INFO:tasks.rados.rados.0.trial039.stdout:327: seq_num 175 ranges {541225=644694,1907290=448390} 2026-02-25T08:43:04.061 INFO:tasks.rados.rados.0.trial039.stdout:327: writing trial03942481-27 from 541225 to 1185919 tid 1 2026-02-25T08:43:04.066 INFO:tasks.rados.rados.0.trial039.stdout:327: writing trial03942481-27 from 1907290 to 2355680 tid 2 2026-02-25T08:43:04.066 INFO:tasks.rados.rados.0.trial039.stdout:328: read oid 23 snap -1 2026-02-25T08:43:04.067 INFO:tasks.rados.rados.0.trial039.stdout:328: expect (ObjNum 155 snap 29 seq_num 155) 2026-02-25T08:43:04.067 INFO:tasks.rados.rados.0.trial039.stdout:329: write oid 43 current snap is 40 2026-02-25T08:43:04.067 INFO:tasks.rados.rados.0.trial039.stdout:329: seq_num 176 ranges {558032=719943,1849419=592048} 2026-02-25T08:43:04.072 INFO:tasks.rados.rados.0.trial039.stdout:329: writing trial03942481-43 from 558032 to 1277975 tid 1 2026-02-25T08:43:04.076 INFO:tasks.rados.rados.0.trial039.stdout:329: writing trial03942481-43 from 1849419 to 2441467 tid 2 2026-02-25T08:43:04.077 INFO:tasks.rados.rados.0.trial039.stdout:330: write oid 31 current snap is 40 2026-02-25T08:43:04.077 INFO:tasks.rados.rados.0.trial039.stdout:325: expect (ObjNum 165 snap 32 seq_num 165) 2026-02-25T08:43:04.079 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1386340 2026-02-25T08:43:04.118 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:04.143 INFO:tasks.rados.rados.0.trial039.stdout:328: expect (ObjNum 155 snap 29 seq_num 155) 2026-02-25T08:43:04.143 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:04.236 INFO:tasks.rados.rados.0.trial039.stdout:329: finishing write tid 1 to trial03942481-43 2026-02-25T08:43:04.237 INFO:tasks.rados.rados.0.trial039.stdout:330: seq_num 177 ranges {574839=795192,1907901=452059} 2026-02-25T08:43:04.242 INFO:tasks.rados.rados.0.trial039.stdout:330: writing trial03942481-31 from 574839 to 1370031 tid 1 2026-02-25T08:43:04.247 INFO:tasks.rados.rados.0.trial039.stdout:330: writing trial03942481-31 from 1907901 to 2359960 tid 2 2026-02-25T08:43:04.247 INFO:tasks.rados.rados.0.trial039.stdout:325: done (4 left) 2026-02-25T08:43:04.247 INFO:tasks.rados.rados.0.trial039.stdout:328: done (3 left) 2026-02-25T08:43:04.247 INFO:tasks.rados.rados.0.trial039.stdout:331: read oid 47 snap -1 2026-02-25T08:43:04.247 INFO:tasks.rados.rados.0.trial039.stdout:331: expect (ObjNum 80 snap 8 seq_num 80) 2026-02-25T08:43:04.247 INFO:tasks.rados.rados.0.trial039.stdout:332: read oid 32 snap 35 2026-02-25T08:43:04.247 INFO:tasks.rados.rados.0.trial039.stdout:332: expect (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:43:04.247 INFO:tasks.rados.rados.0.trial039.stdout:333: read oid 6 snap 39 2026-02-25T08:43:04.247 INFO:tasks.rados.rados.0.trial039.stdout:333: expect (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:334: write oid 49 current snap is 40 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:327: finishing write tid 1 to trial03942481-27 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:329: finishing write tid 2 to trial03942481-43 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:327: finishing write tid 2 to trial03942481-27 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:329: finishing write tid 3 to trial03942481-43 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:329: finishing write tid 4 to trial03942481-43 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:329: oid 43 updating version 0 to 51 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:329: oid 43 version 51 is already newer than 51 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:329: oid 43 version 51 is already newer than 51 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 43 v 51 (ObjNum 176 snap 40 seq_num 176) dirty exists 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:329: left oid 43 (ObjNum 176 snap 40 seq_num 176) 2026-02-25T08:43:04.248 INFO:tasks.rados.rados.0.trial039.stdout:327: finishing write tid 3 to trial03942481-27 2026-02-25T08:43:04.249 INFO:tasks.rados.rados.0.trial039.stdout:327: finishing write tid 4 to trial03942481-27 2026-02-25T08:43:04.249 INFO:tasks.rados.rados.0.trial039.stdout:327: oid 27 updating version 0 to 54 2026-02-25T08:43:04.249 INFO:tasks.rados.rados.0.trial039.stdout:327: oid 27 version 54 is already newer than 54 2026-02-25T08:43:04.249 INFO:tasks.rados.rados.0.trial039.stdout:327: oid 27 version 54 is already newer than 54 2026-02-25T08:43:04.249 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 27 v 54 (ObjNum 175 snap 40 seq_num 175) dirty exists 2026-02-25T08:43:04.249 INFO:tasks.rados.rados.0.trial039.stdout:327: left oid 27 (ObjNum 175 snap 40 seq_num 175) 2026-02-25T08:43:04.249 INFO:tasks.rados.rados.0.trial039.stdout:334: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2026-02-25T08:43:04.250 INFO:tasks.rados.rados.0.trial039.stdout:334: writing trial03942481-49 from 591646 to 1062087 tid 1 2026-02-25T08:43:04.254 INFO:tasks.rados.rados.0.trial039.stdout:334: writing trial03942481-49 from 1566383 to 2162100 tid 2 2026-02-25T08:43:04.258 INFO:tasks.rados.rados.0.trial039.stdout:334: writing trial03942481-49 from 2888822 to 2991646 tid 3 2026-02-25T08:43:04.258 INFO:tasks.rados.rados.0.trial039.stdout:327: done (6 left) 2026-02-25T08:43:04.258 INFO:tasks.rados.rados.0.trial039.stdout:329: done (5 left) 2026-02-25T08:43:04.258 INFO:tasks.rados.rados.0.trial039.stdout:335: read oid 38 snap -1 2026-02-25T08:43:04.258 INFO:tasks.rados.rados.0.trial039.stdout:335: expect (ObjNum 99 snap 11 seq_num 99) 2026-02-25T08:43:04.258 INFO:tasks.rados.rados.0.trial039.stdout:336: delete oid 7 current snap is 40 2026-02-25T08:43:04.263 INFO:tasks.rados.rados.0.trial039.stdout:331: expect (ObjNum 80 snap 8 seq_num 80) 2026-02-25T08:43:04.263 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:04.324 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 3117484 2026-02-25T08:43:04.344 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2514091 2026-02-25T08:43:04.355 INFO:tasks.rados.rados.0.trial039.stdout:333: expect (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:43:04.359 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1002825 2026-02-25T08:43:04.369 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:04.401 INFO:tasks.rados.rados.0.trial039.stdout:335: expect (ObjNum 99 snap 11 seq_num 99) 2026-02-25T08:43:04.412 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2123433 2026-02-25T08:43:04.516 INFO:tasks.rados.rados.0.trial039.stdout:332: expect (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:43:04.516 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:04.535 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 86739 2026-02-25T08:43:04.553 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:04.572 INFO:tasks.rados.rados.0.trial039.stdout:330: finishing write tid 1 to trial03942481-31 2026-02-25T08:43:04.573 INFO:tasks.rados.rados.0.trial039.stdout:330: finishing write tid 2 to trial03942481-31 2026-02-25T08:43:04.573 INFO:tasks.rados.rados.0.trial039.stdout:331: done (6 left) 2026-02-25T08:43:04.573 INFO:tasks.rados.rados.0.trial039.stdout:332: done (5 left) 2026-02-25T08:43:04.573 INFO:tasks.rados.rados.0.trial039.stdout:333: done (4 left) 2026-02-25T08:43:04.573 INFO:tasks.rados.rados.0.trial039.stdout:335: done (3 left) 2026-02-25T08:43:04.573 INFO:tasks.rados.rados.0.trial039.stdout:336: done (2 left) 2026-02-25T08:43:04.573 INFO:tasks.rados.rados.0.trial039.stdout:337: write oid 45 current snap is 40 2026-02-25T08:43:04.573 INFO:tasks.rados.rados.0.trial039.stdout:330: finishing write tid 3 to trial03942481-31 2026-02-25T08:43:04.573 INFO:tasks.rados.rados.0.trial039.stdout:337: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2026-02-25T08:43:04.577 INFO:tasks.rados.rados.0.trial039.stdout:337: writing trial03942481-45 from 625260 to 1246199 tid 1 2026-02-25T08:43:04.581 INFO:tasks.rados.rados.0.trial039.stdout:337: writing trial03942481-45 from 1683347 to 2282733 tid 2 2026-02-25T08:43:04.585 INFO:tasks.rados.rados.0.trial039.stdout:337: writing trial03942481-45 from 2943668 to 3025260 tid 3 2026-02-25T08:43:04.585 INFO:tasks.rados.rados.0.trial039.stdout:338: read oid 4 snap -1 2026-02-25T08:43:04.585 INFO:tasks.rados.rados.0.trial039.stdout:338: expect (ObjNum 144 snap 27 seq_num 144) 2026-02-25T08:43:04.585 INFO:tasks.rados.rados.0.trial039.stdout:339: delete oid 18 current snap is 40 2026-02-25T08:43:04.585 INFO:tasks.rados.rados.0.trial039.stdout:330: finishing write tid 4 to trial03942481-31 2026-02-25T08:43:04.585 INFO:tasks.rados.rados.0.trial039.stdout:330: oid 31 updating version 0 to 102 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:330: oid 31 version 102 is already newer than 102 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:330: oid 31 version 102 is already newer than 102 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 102 (ObjNum 177 snap 40 seq_num 177) dirty exists 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:330: left oid 31 (ObjNum 177 snap 40 seq_num 177) 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:334: finishing write tid 1 to trial03942481-49 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:334: finishing write tid 2 to trial03942481-49 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:334: finishing write tid 3 to trial03942481-49 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:334: finishing write tid 4 to trial03942481-49 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:334: finishing write tid 5 to trial03942481-49 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:334: oid 49 updating version 0 to 59 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:334: oid 49 version 59 is already newer than 59 2026-02-25T08:43:04.586 INFO:tasks.rados.rados.0.trial039.stdout:334: oid 49 version 59 is already newer than 59 2026-02-25T08:43:04.587 INFO:tasks.rados.rados.0.trial039.stdout:334: oid 49 version 59 is already newer than 59 2026-02-25T08:43:04.587 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 49 v 59 (ObjNum 178 snap 40 seq_num 178) dirty exists 2026-02-25T08:43:04.587 INFO:tasks.rados.rados.0.trial039.stdout:334: left oid 49 (ObjNum 178 snap 40 seq_num 178) 2026-02-25T08:43:04.608 INFO:tasks.rados.rados.0.trial039.stdout:338: expect (ObjNum 144 snap 27 seq_num 144) 2026-02-25T08:43:04.613 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1033944 2026-02-25T08:43:04.687 INFO:tasks.rados.rados.0.trial039.stdout:337: finishing write tid 1 to trial03942481-45 2026-02-25T08:43:04.688 INFO:tasks.rados.rados.0.trial039.stdout:337: finishing write tid 2 to trial03942481-45 2026-02-25T08:43:04.688 INFO:tasks.rados.rados.0.trial039.stdout:337: finishing write tid 3 to trial03942481-45 2026-02-25T08:43:04.688 INFO:tasks.rados.rados.0.trial039.stdout:337: finishing write tid 4 to trial03942481-45 2026-02-25T08:43:04.688 INFO:tasks.rados.rados.0.trial039.stdout:337: finishing write tid 5 to trial03942481-45 2026-02-25T08:43:04.688 INFO:tasks.rados.rados.0.trial039.stdout:337: oid 45 updating version 0 to 85 2026-02-25T08:43:04.688 INFO:tasks.rados.rados.0.trial039.stdout:337: oid 45 version 85 is already newer than 85 2026-02-25T08:43:04.688 INFO:tasks.rados.rados.0.trial039.stdout:337: oid 45 version 85 is already newer than 85 2026-02-25T08:43:04.688 INFO:tasks.rados.rados.0.trial039.stdout:337: oid 45 version 85 is already newer than 85 2026-02-25T08:43:04.688 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 45 v 85 (ObjNum 180 snap 40 seq_num 180) dirty exists 2026-02-25T08:43:04.688 INFO:tasks.rados.rados.0.trial039.stdout:337: left oid 45 (ObjNum 180 snap 40 seq_num 180) 2026-02-25T08:43:04.689 INFO:tasks.rados.rados.0.trial039.stdout:330: done (4 left) 2026-02-25T08:43:04.689 INFO:tasks.rados.rados.0.trial039.stdout:334: done (3 left) 2026-02-25T08:43:04.689 INFO:tasks.rados.rados.0.trial039.stdout:337: done (2 left) 2026-02-25T08:43:04.689 INFO:tasks.rados.rados.0.trial039.stdout:338: done (1 left) 2026-02-25T08:43:04.689 INFO:tasks.rados.rados.0.trial039.stdout:339: done (0 left) 2026-02-25T08:43:04.689 INFO:tasks.rados.rados.0.trial039.stdout:340: snap_remove snap 32 2026-02-25T08:43:05.136 INFO:tasks.rados.rados.0.trial039.stdout:340: done (0 left) 2026-02-25T08:43:05.136 INFO:tasks.rados.rados.0.trial039.stdout:341: delete oid 40 current snap is 40 2026-02-25T08:43:05.139 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:05.179 INFO:tasks.rados.rados.0.trial039.stdout:341: done (0 left) 2026-02-25T08:43:05.180 INFO:tasks.rados.rados.0.trial039.stdout:342: read oid 15 snap 34 2026-02-25T08:43:05.180 INFO:tasks.rados.rados.0.trial039.stdout:342: expect deleted 2026-02-25T08:43:05.180 INFO:tasks.rados.rados.0.trial039.stdout:343: snap_remove snap 37 2026-02-25T08:43:05.210 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:05.213 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:05.624 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:05.700 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:05.696+0000 7fd72ef0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:05.702 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:05.700+0000 7fd72ef0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:05.729 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:05.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:06.138 INFO:tasks.rados.rados.0.trial039.stdout:343: done (1 left) 2026-02-25T08:43:06.138 INFO:tasks.rados.rados.0.trial039.stdout:344: snap_create 2026-02-25T08:43:07.143 INFO:tasks.rados.rados.0.trial039.stdout:342: done (1 left) 2026-02-25T08:43:07.143 INFO:tasks.rados.rados.0.trial039.stdout:344: done (0 left) 2026-02-25T08:43:07.143 INFO:tasks.rados.rados.0.trial039.stdout:345: snap_remove snap 40 2026-02-25T08:43:07.748 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:07.826 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:07.822+0000 7fd4d8b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:07.828 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:07.826+0000 7fd4d8b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:07.855 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:07.871 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:08.144 INFO:tasks.rados.rados.0.trial039.stdout:345: done (0 left) 2026-02-25T08:43:08.144 INFO:tasks.rados.rados.0.trial039.stdout:346: read oid 6 snap -1 2026-02-25T08:43:08.144 INFO:tasks.rados.rados.0.trial039.stdout:346: expect (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:43:08.144 INFO:tasks.rados.rados.0.trial039.stdout:347: snap_remove snap 34 2026-02-25T08:43:08.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~94s 2026-02-25T08:43:09.150 INFO:tasks.rados.rados.0.trial039.stdout:347: done (1 left) 2026-02-25T08:43:09.151 INFO:tasks.rados.rados.0.trial039.stdout:348: read oid 38 snap -1 2026-02-25T08:43:09.151 INFO:tasks.rados.rados.0.trial039.stdout:348: expect (ObjNum 99 snap 11 seq_num 99) 2026-02-25T08:43:09.151 INFO:tasks.rados.rados.0.trial039.stdout:349: snap_remove snap 35 2026-02-25T08:43:09.872 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:09.950 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:09.947+0000 7fbabdccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:09.953 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:09.950+0000 7fbabdccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:09.979 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:09.995 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:09.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:10.151 INFO:tasks.rados.rados.0.trial039.stdout:349: done (2 left) 2026-02-25T08:43:10.152 INFO:tasks.rados.rados.0.trial039.stdout:350: snap_create 2026-02-25T08:43:10.152 INFO:tasks.rados.rados.0.trial039.stdout:346: expect (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:43:10.170 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:10.207 INFO:tasks.rados.rados.0.trial039.stdout:348: expect (ObjNum 99 snap 11 seq_num 99) 2026-02-25T08:43:10.207 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:10.214 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:10.249 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:10.284 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:10.287 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:11.158 INFO:tasks.rados.rados.0.trial039.stdout:346: done (2 left) 2026-02-25T08:43:11.158 INFO:tasks.rados.rados.0.trial039.stdout:348: done (1 left) 2026-02-25T08:43:11.158 INFO:tasks.rados.rados.0.trial039.stdout:350: done (0 left) 2026-02-25T08:43:11.158 INFO:tasks.rados.rados.0.trial039.stdout:351: snap_remove snap 38 2026-02-25T08:43:11.997 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:12.074 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:12.072+0000 7ff32fd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:12.077 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:12.075+0000 7ff32fd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:12.102 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:12.117 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:12.158 INFO:tasks.rados.rados.0.trial039.stdout:351: done (0 left) 2026-02-25T08:43:12.158 INFO:tasks.rados.rados.0.trial039.stdout:352: delete oid 48 current snap is 42 2026-02-25T08:43:12.197 INFO:tasks.rados.rados.0.trial039.stdout:352: done (0 left) 2026-02-25T08:43:12.197 INFO:tasks.rados.rados.0.trial039.stdout:353: read oid 46 snap -1 2026-02-25T08:43:12.197 INFO:tasks.rados.rados.0.trial039.stdout:353: expect deleted 2026-02-25T08:43:12.197 INFO:tasks.rados.rados.0.trial039.stdout:354: write (excl) oid 44 current snap is 42 2026-02-25T08:43:12.197 INFO:tasks.rados.rados.0.trial039.stdout:354: seq_num 184 ranges {692488=638288,1917275=606724} 2026-02-25T08:43:12.202 INFO:tasks.rados.rados.0.trial039.stdout:354: writing trial03942481-44 from 692488 to 1330776 tid 1 2026-02-25T08:43:12.207 INFO:tasks.rados.rados.0.trial039.stdout:354: writing trial03942481-44 from 1917275 to 2523999 tid 2 2026-02-25T08:43:12.207 INFO:tasks.rados.rados.0.trial039.stdout:355: write oid 11 current snap is 42 2026-02-25T08:43:12.207 INFO:tasks.rados.rados.0.trial039.stdout:355: seq_num 185 ranges {709295=713537,1859404=750382} 2026-02-25T08:43:12.212 INFO:tasks.rados.rados.0.trial039.stdout:355: writing trial03942481-11 from 709295 to 1422832 tid 1 2026-02-25T08:43:12.217 INFO:tasks.rados.rados.0.trial039.stdout:355: writing trial03942481-11 from 1859404 to 2609786 tid 2 2026-02-25T08:43:12.217 INFO:tasks.rados.rados.0.trial039.stdout:356: snap_create 2026-02-25T08:43:12.229 INFO:tasks.rados.rados.0.trial039.stdout:354: finishing write tid 1 to trial03942481-44 2026-02-25T08:43:12.244 INFO:tasks.rados.rados.0.trial039.stdout:354: finishing write tid 2 to trial03942481-44 2026-02-25T08:43:12.249 INFO:tasks.rados.rados.0.trial039.stdout:354: finishing write tid 3 to trial03942481-44 2026-02-25T08:43:12.250 INFO:tasks.rados.rados.0.trial039.stdout:355: finishing write tid 1 to trial03942481-11 2026-02-25T08:43:12.251 INFO:tasks.rados.rados.0.trial039.stdout:354: finishing write tid 4 to trial03942481-44 2026-02-25T08:43:12.251 INFO:tasks.rados.rados.0.trial039.stdout:354: oid 44 updating version 0 to 66 2026-02-25T08:43:12.251 INFO:tasks.rados.rados.0.trial039.stdout:354: oid 44 version 66 is already newer than 64 2026-02-25T08:43:12.251 INFO:tasks.rados.rados.0.trial039.stdout:354: oid 44 version 66 is already newer than 66 2026-02-25T08:43:12.252 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 66 (ObjNum 184 snap 42 seq_num 184) dirty exists 2026-02-25T08:43:12.252 INFO:tasks.rados.rados.0.trial039.stdout:354: left oid 44 (ObjNum 184 snap 42 seq_num 184) 2026-02-25T08:43:12.262 INFO:tasks.rados.rados.0.trial039.stdout:355: finishing write tid 2 to trial03942481-11 2026-02-25T08:43:12.266 INFO:tasks.rados.rados.0.trial039.stdout:355: finishing write tid 3 to trial03942481-11 2026-02-25T08:43:12.268 INFO:tasks.rados.rados.0.trial039.stdout:355: finishing write tid 4 to trial03942481-11 2026-02-25T08:43:12.268 INFO:tasks.rados.rados.0.trial039.stdout:355: oid 11 updating version 0 to 16 2026-02-25T08:43:12.268 INFO:tasks.rados.rados.0.trial039.stdout:355: oid 11 version 16 is already newer than 16 2026-02-25T08:43:12.268 INFO:tasks.rados.rados.0.trial039.stdout:355: oid 11 version 16 is already newer than 16 2026-02-25T08:43:12.268 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 11 v 16 (ObjNum 185 snap 42 seq_num 185) dirty exists 2026-02-25T08:43:12.268 INFO:tasks.rados.rados.0.trial039.stdout:355: left oid 11 (ObjNum 185 snap 42 seq_num 185) 2026-02-25T08:43:13.058 INFO:tasks.rados.rados.0.trial039.stdout:353: done (3 left) 2026-02-25T08:43:13.059 INFO:tasks.rados.rados.0.trial039.stdout:354: done (2 left) 2026-02-25T08:43:13.059 INFO:tasks.rados.rados.0.trial039.stdout:355: done (1 left) 2026-02-25T08:43:13.059 INFO:tasks.rados.rados.0.trial039.stdout:356: done (0 left) 2026-02-25T08:43:13.059 INFO:tasks.rados.rados.0.trial039.stdout:357: snap_remove snap 41 2026-02-25T08:43:14.059 INFO:tasks.rados.rados.0.trial039.stdout:357: done (0 left) 2026-02-25T08:43:14.059 INFO:tasks.rados.rados.0.trial039.stdout:358: snap_create 2026-02-25T08:43:14.117 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:14.191 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:14.189+0000 7f62ed851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:14.194 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:14.193+0000 7f62ed851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:14.220 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:14.236 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:14.237 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:14.405 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~99s 2026-02-25T08:43:15.059 INFO:tasks.rados.rados.0.trial039.stdout:358: done (0 left) 2026-02-25T08:43:15.059 INFO:tasks.rados.rados.0.trial039.stdout:359: read oid 32 snap -1 2026-02-25T08:43:15.059 INFO:tasks.rados.rados.0.trial039.stdout:359: expect (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:43:15.059 INFO:tasks.rados.rados.0.trial039.stdout:360: snap_remove snap 42 2026-02-25T08:43:15.289 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:15.362 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:15.365 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:15.365 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:16.060 INFO:tasks.rados.rados.0.trial039.stdout:360: done (1 left) 2026-02-25T08:43:16.061 INFO:tasks.rados.rados.0.trial039.stdout:361: snap_create 2026-02-25T08:43:16.061 INFO:tasks.rados.rados.0.trial039.stdout:359: expect (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:43:16.061 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:16.081 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:16.100 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:16.238 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:16.315 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:16.312+0000 7fb714fec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:16.317 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:16.316+0000 7fb714fec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:16.343 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:16.360 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:17.063 INFO:tasks.rados.rados.0.trial039.stdout:359: done (1 left) 2026-02-25T08:43:17.063 INFO:tasks.rados.rados.0.trial039.stdout:361: done (0 left) 2026-02-25T08:43:17.063 INFO:tasks.rados.rados.0.trial039.stdout:362: read oid 5 snap -1 2026-02-25T08:43:17.063 INFO:tasks.rados.rados.0.trial039.stdout:362: expect (ObjNum 152 snap 29 seq_num 152) 2026-02-25T08:43:17.063 INFO:tasks.rados.rados.0.trial039.stdout:363: snap_create 2026-02-25T08:43:17.083 INFO:tasks.rados.rados.0.trial039.stdout:362: expect (ObjNum 152 snap 29 seq_num 152) 2026-02-25T08:43:17.089 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2443397 2026-02-25T08:43:17.094 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:18.067 INFO:tasks.rados.rados.0.trial039.stdout:362: done (1 left) 2026-02-25T08:43:18.067 INFO:tasks.rados.rados.0.trial039.stdout:363: done (0 left) 2026-02-25T08:43:18.067 INFO:tasks.rados.rados.0.trial039.stdout:364: read oid 37 snap -1 2026-02-25T08:43:18.068 INFO:tasks.rados.rados.0.trial039.stdout:364: expect (ObjNum 130 snap 21 seq_num 130) 2026-02-25T08:43:18.068 INFO:tasks.rados.rados.0.trial039.stdout:365: write (excl) oid 44 current snap is 46 2026-02-25T08:43:18.068 INFO:tasks.rados.rados.0.trial039.stdout:365: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2026-02-25T08:43:18.072 INFO:tasks.rados.rados.0.trial039.stdout:365: writing trial03942481-44 from 726102 to 1514888 tid 1 2026-02-25T08:43:18.076 INFO:tasks.rados.rados.0.trial039.stdout:365: writing trial03942481-44 from 1917886 to 2528279 tid 2 2026-02-25T08:43:18.079 INFO:tasks.rados.rados.0.trial039.stdout:365: writing trial03942481-44 from 3108206 to 3126102 tid 3 2026-02-25T08:43:18.079 INFO:tasks.rados.rados.0.trial039.stdout:366: read oid 26 snap -1 2026-02-25T08:43:18.079 INFO:tasks.rados.rados.0.trial039.stdout:366: expect (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:43:18.079 INFO:tasks.rados.rados.0.trial039.stdout:367: snap_remove snap 44 2026-02-25T08:43:18.361 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:18.437 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:18.435+0000 7f74c488b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:18.440 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:18.438+0000 7f74c488b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:18.465 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:18.482 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:18.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:19.073 INFO:tasks.rados.rados.0.trial039.stdout:367: done (3 left) 2026-02-25T08:43:19.073 INFO:tasks.rados.rados.0.trial039.stdout:368: write oid 21 current snap is 46 2026-02-25T08:43:19.073 INFO:tasks.rados.rados.0.trial039.stdout:368: seq_num 187 ranges {742909=464035,1976368=754051} 2026-02-25T08:43:19.077 INFO:tasks.rados.rados.0.trial039.stdout:368: writing trial03942481-21 from 742909 to 1206944 tid 1 2026-02-25T08:43:19.084 INFO:tasks.rados.rados.0.trial039.stdout:368: writing trial03942481-21 from 1976368 to 2730419 tid 2 2026-02-25T08:43:19.084 INFO:tasks.rados.rados.0.trial039.stdout:369: read oid 4 snap -1 2026-02-25T08:43:19.084 INFO:tasks.rados.rados.0.trial039.stdout:369: expect (ObjNum 144 snap 27 seq_num 144) 2026-02-25T08:43:19.084 INFO:tasks.rados.rados.0.trial039.stdout:370: write (excl) oid 6 current snap is 46 2026-02-25T08:43:19.084 INFO:tasks.rados.rados.0.trial039.stdout:370: seq_num 188 ranges {759716=539284,2034850=614062} 2026-02-25T08:43:19.089 INFO:tasks.rados.rados.0.trial039.stdout:370: writing trial03942481-6 from 759716 to 1299000 tid 1 2026-02-25T08:43:19.095 INFO:tasks.rados.rados.0.trial039.stdout:370: writing trial03942481-6 from 2034850 to 2648912 tid 2 2026-02-25T08:43:19.096 INFO:tasks.rados.rados.0.trial039.stdout:371: delete oid 49 current snap is 46 2026-02-25T08:43:19.096 INFO:tasks.rados.rados.0.trial039.stdout:366: expect (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:43:19.114 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:19.159 INFO:tasks.rados.rados.0.trial039.stdout:364: expect (ObjNum 130 snap 21 seq_num 130) 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:365: finishing write tid 1 to trial03942481-44 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:365: finishing write tid 2 to trial03942481-44 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:364: done (6 left) 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:366: done (5 left) 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:371: done (4 left) 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:372: read oid 49 snap 39 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:372: expect (ObjNum 108 snap 12 seq_num 108) 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:373: read oid 26 snap 45 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:373: expect (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:374: snap_create 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:365: finishing write tid 3 to trial03942481-44 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:365: finishing write tid 4 to trial03942481-44 2026-02-25T08:43:19.216 INFO:tasks.rados.rados.0.trial039.stdout:365: finishing write tid 5 to trial03942481-44 2026-02-25T08:43:19.217 INFO:tasks.rados.rados.0.trial039.stdout:365: oid 44 updating version 0 to 71 2026-02-25T08:43:19.217 INFO:tasks.rados.rados.0.trial039.stdout:365: oid 44 version 71 is already newer than 71 2026-02-25T08:43:19.217 INFO:tasks.rados.rados.0.trial039.stdout:365: oid 44 version 71 is already newer than 71 2026-02-25T08:43:19.217 INFO:tasks.rados.rados.0.trial039.stdout:365: oid 44 version 71 is already newer than 68 2026-02-25T08:43:19.217 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 71 (ObjNum 186 snap 46 seq_num 186) dirty exists 2026-02-25T08:43:19.217 INFO:tasks.rados.rados.0.trial039.stdout:365: left oid 44 (ObjNum 186 snap 46 seq_num 186) 2026-02-25T08:43:19.217 INFO:tasks.rados.rados.0.trial039.stdout:369: expect (ObjNum 144 snap 27 seq_num 144) 2026-02-25T08:43:19.255 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2095631 2026-02-25T08:43:19.262 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:19.291 INFO:tasks.rados.rados.0.trial039.stdout:368: finishing write tid 1 to trial03942481-21 2026-02-25T08:43:19.291 INFO:tasks.rados.rados.0.trial039.stdout:368: finishing write tid 2 to trial03942481-21 2026-02-25T08:43:19.291 INFO:tasks.rados.rados.0.trial039.stdout:368: finishing write tid 3 to trial03942481-21 2026-02-25T08:43:19.291 INFO:tasks.rados.rados.0.trial039.stdout:368: finishing write tid 4 to trial03942481-21 2026-02-25T08:43:19.291 INFO:tasks.rados.rados.0.trial039.stdout:368: oid 21 updating version 0 to 105 2026-02-25T08:43:19.291 INFO:tasks.rados.rados.0.trial039.stdout:368: oid 21 version 105 is already newer than 105 2026-02-25T08:43:19.291 INFO:tasks.rados.rados.0.trial039.stdout:368: oid 21 version 105 is already newer than 105 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 21 v 105 (ObjNum 187 snap 46 seq_num 187) dirty exists 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:368: left oid 21 (ObjNum 187 snap 46 seq_num 187) 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:370: finishing write tid 1 to trial03942481-6 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:370: finishing write tid 2 to trial03942481-6 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:370: finishing write tid 3 to trial03942481-6 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:370: finishing write tid 4 to trial03942481-6 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:370: oid 6 updating version 0 to 109 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:370: oid 6 version 109 is already newer than 109 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:370: oid 6 version 109 is already newer than 107 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 6 v 109 (ObjNum 188 snap 46 seq_num 188) dirty exists 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:370: left oid 6 (ObjNum 188 snap 46 seq_num 188) 2026-02-25T08:43:19.292 INFO:tasks.rados.rados.0.trial039.stdout:373: expect (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:43:19.298 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1336551 2026-02-25T08:43:19.316 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1183666 2026-02-25T08:43:19.327 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:19.351 INFO:tasks.rados.rados.0.trial039.stdout:372: expect (ObjNum 108 snap 12 seq_num 108) 2026-02-25T08:43:19.351 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:19.397 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:19.449 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1584593 2026-02-25T08:43:19.908 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~105s 2026-02-25T08:43:20.077 INFO:tasks.rados.rados.0.trial039.stdout:365: done (6 left) 2026-02-25T08:43:20.077 INFO:tasks.rados.rados.0.trial039.stdout:368: done (5 left) 2026-02-25T08:43:20.077 INFO:tasks.rados.rados.0.trial039.stdout:369: done (4 left) 2026-02-25T08:43:20.077 INFO:tasks.rados.rados.0.trial039.stdout:370: done (3 left) 2026-02-25T08:43:20.077 INFO:tasks.rados.rados.0.trial039.stdout:372: done (2 left) 2026-02-25T08:43:20.077 INFO:tasks.rados.rados.0.trial039.stdout:373: done (1 left) 2026-02-25T08:43:20.077 INFO:tasks.rados.rados.0.trial039.stdout:374: done (0 left) 2026-02-25T08:43:20.077 INFO:tasks.rados.rados.0.trial039.stdout:375: write (excl) oid 33 current snap is 47 2026-02-25T08:43:20.077 INFO:tasks.rados.rados.0.trial039.stdout:375: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2026-02-25T08:43:20.083 INFO:tasks.rados.rados.0.trial039.stdout:375: writing trial03942481-33 from 793330 to 1483112 tid 1 2026-02-25T08:43:20.088 INFO:tasks.rados.rados.0.trial039.stdout:375: writing trial03942481-33 from 2035461 to 2653192 tid 2 2026-02-25T08:43:20.090 INFO:tasks.rados.rados.0.trial039.stdout:375: writing trial03942481-33 from 3101545 to 3193330 tid 3 2026-02-25T08:43:20.090 INFO:tasks.rados.rados.0.trial039.stdout:376: write oid 9 current snap is 47 2026-02-25T08:43:20.090 INFO:tasks.rados.rados.0.trial039.stdout:376: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2026-02-25T08:43:20.093 INFO:tasks.rados.rados.0.trial039.stdout:376: writing trial03942481-9 from 410137 to 891521 tid 1 2026-02-25T08:43:20.098 INFO:tasks.rados.rados.0.trial039.stdout:376: writing trial03942481-9 from 1410296 to 2171685 tid 2 2026-02-25T08:43:20.102 INFO:tasks.rados.rados.0.trial039.stdout:376: writing trial03942481-9 from 2728968 to 3210137 tid 3 2026-02-25T08:43:20.102 INFO:tasks.rados.rados.0.trial039.stdout:377: write (excl) oid 10 current snap is 47 2026-02-25T08:43:20.102 INFO:tasks.rados.rados.0.trial039.stdout:377: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2026-02-25T08:43:20.106 INFO:tasks.rados.rados.0.trial039.stdout:377: writing trial03942481-10 from 426944 to 983577 tid 1 2026-02-25T08:43:20.110 INFO:tasks.rados.rados.0.trial039.stdout:377: writing trial03942481-10 from 1468778 to 2090178 tid 2 2026-02-25T08:43:20.114 INFO:tasks.rados.rados.0.trial039.stdout:377: writing trial03942481-10 from 2872744 to 3226944 tid 3 2026-02-25T08:43:20.114 INFO:tasks.rados.rados.0.trial039.stdout:378: write (excl) oid 18 current snap is 47 2026-02-25T08:43:20.114 INFO:tasks.rados.rados.0.trial039.stdout:378: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2026-02-25T08:43:20.119 INFO:tasks.rados.rados.0.trial039.stdout:378: writing trial03942481-18 from 443751 to 1075633 tid 1 2026-02-25T08:43:20.124 INFO:tasks.rados.rados.0.trial039.stdout:378: writing trial03942481-18 from 1810907 to 2575965 tid 2 2026-02-25T08:43:20.127 INFO:tasks.rados.rados.0.trial039.stdout:378: writing trial03942481-18 from 3067461 to 3243751 tid 3 2026-02-25T08:43:20.127 INFO:tasks.rados.rados.0.trial039.stdout:379: snap_remove snap 46 2026-02-25T08:43:20.366 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:20.437 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:20.440 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:20.441 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:20.484 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:20.562 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:20.559+0000 7fcc7b75b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:20.564 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:20.563+0000 7fcc7b75b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:20.591 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:20.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:21.082 INFO:tasks.rados.rados.0.trial039.stdout:379: done (4 left) 2026-02-25T08:43:21.082 INFO:tasks.rados.rados.0.trial039.stdout:380: read oid 30 snap 39 2026-02-25T08:43:21.082 INFO:tasks.rados.rados.0.trial039.stdout:380: expect (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:43:21.082 INFO:tasks.rados.rados.0.trial039.stdout:381: read oid 22 snap 45 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:381: expect (ObjNum 145 snap 27 seq_num 145) 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:382: snap_create 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:376: finishing write tid 1 to trial03942481-9 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:376: finishing write tid 2 to trial03942481-9 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:378: finishing write tid 1 to trial03942481-18 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:376: finishing write tid 3 to trial03942481-9 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:378: finishing write tid 2 to trial03942481-18 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:376: finishing write tid 4 to trial03942481-9 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:378: finishing write tid 3 to trial03942481-18 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:376: finishing write tid 5 to trial03942481-9 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:376: oid 9 updating version 0 to 19 2026-02-25T08:43:21.083 INFO:tasks.rados.rados.0.trial039.stdout:376: oid 9 version 19 is already newer than 19 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:376: oid 9 version 19 is already newer than 19 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:376: oid 9 version 19 is already newer than 19 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 9 v 19 (ObjNum 191 snap 47 seq_num 191) dirty exists 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:376: left oid 9 (ObjNum 191 snap 47 seq_num 191) 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:378: finishing write tid 4 to trial03942481-18 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:378: finishing write tid 5 to trial03942481-18 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:378: oid 18 updating version 0 to 67 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:378: oid 18 version 67 is already newer than 67 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:378: oid 18 version 67 is already newer than 64 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:378: oid 18 version 67 is already newer than 67 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 18 v 67 (ObjNum 193 snap 47 seq_num 193) dirty exists 2026-02-25T08:43:21.084 INFO:tasks.rados.rados.0.trial039.stdout:378: left oid 18 (ObjNum 193 snap 47 seq_num 193) 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:375: finishing write tid 1 to trial03942481-33 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:375: finishing write tid 2 to trial03942481-33 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:375: finishing write tid 3 to trial03942481-33 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:375: finishing write tid 4 to trial03942481-33 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:375: finishing write tid 5 to trial03942481-33 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:375: oid 33 updating version 0 to 56 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:375: oid 33 version 56 is already newer than 56 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:375: oid 33 version 56 is already newer than 56 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:375: oid 33 version 56 is already newer than 53 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 33 v 56 (ObjNum 190 snap 47 seq_num 190) dirty exists 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:375: left oid 33 (ObjNum 190 snap 47 seq_num 190) 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:377: finishing write tid 1 to trial03942481-10 2026-02-25T08:43:21.085 INFO:tasks.rados.rados.0.trial039.stdout:377: finishing write tid 2 to trial03942481-10 2026-02-25T08:43:21.086 INFO:tasks.rados.rados.0.trial039.stdout:377: finishing write tid 3 to trial03942481-10 2026-02-25T08:43:21.086 INFO:tasks.rados.rados.0.trial039.stdout:377: finishing write tid 4 to trial03942481-10 2026-02-25T08:43:21.086 INFO:tasks.rados.rados.0.trial039.stdout:377: finishing write tid 5 to trial03942481-10 2026-02-25T08:43:21.086 INFO:tasks.rados.rados.0.trial039.stdout:377: oid 10 updating version 0 to 58 2026-02-25T08:43:21.086 INFO:tasks.rados.rados.0.trial039.stdout:377: oid 10 updating version 58 to 61 2026-02-25T08:43:21.086 INFO:tasks.rados.rados.0.trial039.stdout:377: oid 10 version 61 is already newer than 61 2026-02-25T08:43:21.086 INFO:tasks.rados.rados.0.trial039.stdout:377: oid 10 version 61 is already newer than 61 2026-02-25T08:43:21.086 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 10 v 61 (ObjNum 192 snap 47 seq_num 192) dirty exists 2026-02-25T08:43:21.086 INFO:tasks.rados.rados.0.trial039.stdout:377: left oid 10 (ObjNum 192 snap 47 seq_num 192) 2026-02-25T08:43:21.110 INFO:tasks.rados.rados.0.trial039.stdout:381: expect (ObjNum 145 snap 27 seq_num 145) 2026-02-25T08:43:21.193 INFO:tasks.rados.rados.0.trial039.stdout:380: expect (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:43:22.086 INFO:tasks.rados.rados.0.trial039.stdout:375: done (6 left) 2026-02-25T08:43:22.086 INFO:tasks.rados.rados.0.trial039.stdout:376: done (5 left) 2026-02-25T08:43:22.086 INFO:tasks.rados.rados.0.trial039.stdout:377: done (4 left) 2026-02-25T08:43:22.086 INFO:tasks.rados.rados.0.trial039.stdout:378: done (3 left) 2026-02-25T08:43:22.086 INFO:tasks.rados.rados.0.trial039.stdout:380: done (2 left) 2026-02-25T08:43:22.086 INFO:tasks.rados.rados.0.trial039.stdout:381: done (1 left) 2026-02-25T08:43:22.087 INFO:tasks.rados.rados.0.trial039.stdout:382: done (0 left) 2026-02-25T08:43:22.087 INFO:tasks.rados.rados.0.trial039.stdout:383: read oid 6 snap 43 2026-02-25T08:43:22.087 INFO:tasks.rados.rados.0.trial039.stdout:383: expect (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:43:22.087 INFO:tasks.rados.rados.0.trial039.stdout:384: snap_create 2026-02-25T08:43:22.107 INFO:tasks.rados.rados.0.trial039.stdout:383: expect (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:43:22.107 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:22.610 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:22.685 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:22.683+0000 7fd442ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:22.688 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:22.687+0000 7fd442ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:22.713 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:22.731 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:23.061 INFO:tasks.rados.rados.0.trial039.stdout:383: done (1 left) 2026-02-25T08:43:23.061 INFO:tasks.rados.rados.0.trial039.stdout:384: done (0 left) 2026-02-25T08:43:23.061 INFO:tasks.rados.rados.0.trial039.stdout:385: delete oid 23 current snap is 49 2026-02-25T08:43:23.101 INFO:tasks.rados.rados.0.trial039.stdout:385: done (0 left) 2026-02-25T08:43:23.101 INFO:tasks.rados.rados.0.trial039.stdout:386: read oid 31 snap -1 2026-02-25T08:43:23.101 INFO:tasks.rados.rados.0.trial039.stdout:386: expect (ObjNum 177 snap 40 seq_num 177) 2026-02-25T08:43:23.101 INFO:tasks.rados.rados.0.trial039.stdout:387: read oid 32 snap 47 2026-02-25T08:43:23.101 INFO:tasks.rados.rados.0.trial039.stdout:387: expect (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:43:23.101 INFO:tasks.rados.rados.0.trial039.stdout:388: delete oid 22 current snap is 49 2026-02-25T08:43:23.110 INFO:tasks.rados.rados.0.trial039.stdout:387: expect (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:43:23.163 INFO:tasks.rados.rados.0.trial039.stdout:386: expect (ObjNum 177 snap 40 seq_num 177) 2026-02-25T08:43:23.163 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:23.188 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:23.213 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 8238 2026-02-25T08:43:23.239 INFO:tasks.rados.rados.0.trial039.stdout:386: done (2 left) 2026-02-25T08:43:23.239 INFO:tasks.rados.rados.0.trial039.stdout:387: done (1 left) 2026-02-25T08:43:23.239 INFO:tasks.rados.rados.0.trial039.stdout:388: done (0 left) 2026-02-25T08:43:23.239 INFO:tasks.rados.rados.0.trial039.stdout:389: read oid 21 snap -1 2026-02-25T08:43:23.240 INFO:tasks.rados.rados.0.trial039.stdout:389: expect (ObjNum 187 snap 46 seq_num 187) 2026-02-25T08:43:23.240 INFO:tasks.rados.rados.0.trial039.stdout:390: read oid 32 snap 48 2026-02-25T08:43:23.240 INFO:tasks.rados.rados.0.trial039.stdout:390: expect (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:43:23.240 INFO:tasks.rados.rados.0.trial039.stdout:391: write (excl) oid 24 current snap is 49 2026-02-25T08:43:23.240 INFO:tasks.rados.rados.0.trial039.stdout:391: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2026-02-25T08:43:23.242 INFO:tasks.rados.rados.0.trial039.stdout:391: writing trial03942481-24 from 494172 to 951801 tid 1 2026-02-25T08:43:23.247 INFO:tasks.rados.rados.0.trial039.stdout:391: writing trial03942481-24 from 1586353 to 2098738 tid 2 2026-02-25T08:43:23.252 INFO:tasks.rados.rados.0.trial039.stdout:391: writing trial03942481-24 from 2749730 to 3294172 tid 3 2026-02-25T08:43:23.252 INFO:tasks.rados.rados.0.trial039.stdout:392: read oid 2 snap -1 2026-02-25T08:43:23.252 INFO:tasks.rados.rados.0.trial039.stdout:392: expect deleted 2026-02-25T08:43:23.252 INFO:tasks.rados.rados.0.trial039.stdout:393: write oid 8 current snap is 49 2026-02-25T08:43:23.252 INFO:tasks.rados.rados.0.trial039.stdout:393: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2026-02-25T08:43:23.255 INFO:tasks.rados.rados.0.trial039.stdout:393: writing trial03942481-8 from 510979 to 1043857 tid 1 2026-02-25T08:43:23.260 INFO:tasks.rados.rados.0.trial039.stdout:393: writing trial03942481-8 from 1528482 to 2300878 tid 2 2026-02-25T08:43:23.264 INFO:tasks.rados.rados.0.trial039.stdout:393: writing trial03942481-8 from 3060800 to 3310979 tid 3 2026-02-25T08:43:23.265 INFO:tasks.rados.rados.0.trial039.stdout:394: read oid 44 snap -1 2026-02-25T08:43:23.265 INFO:tasks.rados.rados.0.trial039.stdout:394: expect (ObjNum 186 snap 46 seq_num 186) 2026-02-25T08:43:23.265 INFO:tasks.rados.rados.0.trial039.stdout:395: delete oid 9 current snap is 49 2026-02-25T08:43:23.265 INFO:tasks.rados.rados.0.trial039.stdout:389: expect (ObjNum 187 snap 46 seq_num 187) 2026-02-25T08:43:23.265 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 315248 2026-02-25T08:43:23.316 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:23.343 INFO:tasks.rados.rados.0.trial039.stdout:389: done (6 left) 2026-02-25T08:43:23.343 INFO:tasks.rados.rados.0.trial039.stdout:395: done (5 left) 2026-02-25T08:43:23.343 INFO:tasks.rados.rados.0.trial039.stdout:396: snap_remove snap 43 2026-02-25T08:43:24.090 INFO:tasks.rados.rados.0.trial039.stdout:396: done (5 left) 2026-02-25T08:43:24.090 INFO:tasks.rados.rados.0.trial039.stdout:397: snap_create 2026-02-25T08:43:24.090 INFO:tasks.rados.rados.0.trial039.stdout:390: expect (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:43:24.090 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:24.129 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:24.147 INFO:tasks.rados.rados.0.trial039.stdout:391: finishing write tid 1 to trial03942481-24 2026-02-25T08:43:24.148 INFO:tasks.rados.rados.0.trial039.stdout:393: finishing write tid 1 to trial03942481-8 2026-02-25T08:43:24.148 INFO:tasks.rados.rados.0.trial039.stdout:394: expect (ObjNum 186 snap 46 seq_num 186) 2026-02-25T08:43:24.178 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2944056 2026-02-25T08:43:24.215 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2151585 2026-02-25T08:43:24.283 INFO:tasks.rados.rados.0.trial039.stdout:393: finishing write tid 2 to trial03942481-8 2026-02-25T08:43:24.284 INFO:tasks.rados.rados.0.trial039.stdout:391: finishing write tid 2 to trial03942481-24 2026-02-25T08:43:24.284 INFO:tasks.rados.rados.0.trial039.stdout:393: finishing write tid 3 to trial03942481-8 2026-02-25T08:43:24.284 INFO:tasks.rados.rados.0.trial039.stdout:391: finishing write tid 3 to trial03942481-24 2026-02-25T08:43:24.284 INFO:tasks.rados.rados.0.trial039.stdout:393: finishing write tid 4 to trial03942481-8 2026-02-25T08:43:24.284 INFO:tasks.rados.rados.0.trial039.stdout:391: finishing write tid 4 to trial03942481-24 2026-02-25T08:43:24.284 INFO:tasks.rados.rados.0.trial039.stdout:393: finishing write tid 5 to trial03942481-8 2026-02-25T08:43:24.284 INFO:tasks.rados.rados.0.trial039.stdout:393: oid 8 updating version 0 to 65 2026-02-25T08:43:24.284 INFO:tasks.rados.rados.0.trial039.stdout:393: oid 8 version 65 is already newer than 65 2026-02-25T08:43:24.284 INFO:tasks.rados.rados.0.trial039.stdout:393: oid 8 version 65 is already newer than 65 2026-02-25T08:43:24.285 INFO:tasks.rados.rados.0.trial039.stdout:393: oid 8 version 65 is already newer than 65 2026-02-25T08:43:24.285 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 8 v 65 (ObjNum 197 snap 49 seq_num 197) dirty exists 2026-02-25T08:43:24.285 INFO:tasks.rados.rados.0.trial039.stdout:393: left oid 8 (ObjNum 197 snap 49 seq_num 197) 2026-02-25T08:43:24.285 INFO:tasks.rados.rados.0.trial039.stdout:391: finishing write tid 5 to trial03942481-24 2026-02-25T08:43:24.285 INFO:tasks.rados.rados.0.trial039.stdout:391: oid 24 updating version 0 to 39 2026-02-25T08:43:24.285 INFO:tasks.rados.rados.0.trial039.stdout:391: oid 24 version 39 is already newer than 39 2026-02-25T08:43:24.285 INFO:tasks.rados.rados.0.trial039.stdout:391: oid 24 version 39 is already newer than 36 2026-02-25T08:43:24.285 INFO:tasks.rados.rados.0.trial039.stdout:391: oid 24 version 39 is already newer than 39 2026-02-25T08:43:24.285 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 24 v 39 (ObjNum 196 snap 49 seq_num 196) dirty exists 2026-02-25T08:43:24.285 INFO:tasks.rados.rados.0.trial039.stdout:391: left oid 24 (ObjNum 196 snap 49 seq_num 196) 2026-02-25T08:43:24.732 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:24.810 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:24.807+0000 7fbb5e59c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:24.812 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:24.811+0000 7fbb5e59c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:24.837 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:24.853 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:24.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:25.094 INFO:tasks.rados.rados.0.trial039.stdout:390: done (5 left) 2026-02-25T08:43:25.095 INFO:tasks.rados.rados.0.trial039.stdout:391: done (4 left) 2026-02-25T08:43:25.095 INFO:tasks.rados.rados.0.trial039.stdout:392: done (3 left) 2026-02-25T08:43:25.095 INFO:tasks.rados.rados.0.trial039.stdout:393: done (2 left) 2026-02-25T08:43:25.095 INFO:tasks.rados.rados.0.trial039.stdout:394: done (1 left) 2026-02-25T08:43:25.095 INFO:tasks.rados.rados.0.trial039.stdout:397: done (0 left) 2026-02-25T08:43:25.095 INFO:tasks.rados.rados.0.trial039.stdout:398: write oid 6 current snap is 50 2026-02-25T08:43:25.095 INFO:tasks.rados.rados.0.trial039.stdout:398: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2026-02-25T08:43:25.099 INFO:tasks.rados.rados.0.trial039.stdout:398: writing trial03942481-6 from 544593 to 1344322 tid 1 2026-02-25T08:43:25.104 INFO:tasks.rados.rados.0.trial039.stdout:398: writing trial03942481-6 from 1761799 to 2537864 tid 2 2026-02-25T08:43:25.107 INFO:tasks.rados.rados.0.trial039.stdout:398: writing trial03942481-6 from 2948352 to 3344593 tid 3 2026-02-25T08:43:25.107 INFO:tasks.rados.rados.0.trial039.stdout:399: snap_create 2026-02-25T08:43:25.145 INFO:tasks.rados.rados.0.trial039.stdout:398: finishing write tid 1 to trial03942481-6 2026-02-25T08:43:25.153 INFO:tasks.rados.rados.0.trial039.stdout:398: finishing write tid 2 to trial03942481-6 2026-02-25T08:43:25.159 INFO:tasks.rados.rados.0.trial039.stdout:398: finishing write tid 3 to trial03942481-6 2026-02-25T08:43:25.163 INFO:tasks.rados.rados.0.trial039.stdout:398: finishing write tid 4 to trial03942481-6 2026-02-25T08:43:25.165 INFO:tasks.rados.rados.0.trial039.stdout:398: finishing write tid 5 to trial03942481-6 2026-02-25T08:43:25.165 INFO:tasks.rados.rados.0.trial039.stdout:398: oid 6 updating version 0 to 116 2026-02-25T08:43:25.166 INFO:tasks.rados.rados.0.trial039.stdout:398: oid 6 version 116 is already newer than 116 2026-02-25T08:43:25.166 INFO:tasks.rados.rados.0.trial039.stdout:398: oid 6 version 116 is already newer than 116 2026-02-25T08:43:25.166 INFO:tasks.rados.rados.0.trial039.stdout:398: oid 6 version 116 is already newer than 116 2026-02-25T08:43:25.166 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 6 v 116 (ObjNum 199 snap 50 seq_num 199) dirty exists 2026-02-25T08:43:25.166 INFO:tasks.rados.rados.0.trial039.stdout:398: left oid 6 (ObjNum 199 snap 50 seq_num 199) 2026-02-25T08:43:25.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~110s 2026-02-25T08:43:25.441 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:25.513 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:25.517 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:26.101 INFO:tasks.rados.rados.0.trial039.stdout:398: done (1 left) 2026-02-25T08:43:26.102 INFO:tasks.rados.rados.0.trial039.stdout:399: done (0 left) 2026-02-25T08:43:26.102 INFO:tasks.rados.rados.0.trial039.stdout:400: write oid 22 current snap is 51 2026-02-25T08:43:26.102 INFO:tasks.rados.rados.0.trial039.stdout:400: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2026-02-25T08:43:26.105 INFO:tasks.rados.rados.0.trial039.stdout:400: writing trial03942481-22 from 561400 to 1036378 tid 1 2026-02-25T08:43:26.110 INFO:tasks.rados.rados.0.trial039.stdout:400: writing trial03942481-22 from 1820281 to 2340004 tid 2 2026-02-25T08:43:26.115 INFO:tasks.rados.rados.0.trial039.stdout:400: writing trial03942481-22 from 2859422 to 3317001 tid 3 2026-02-25T08:43:26.115 INFO:tasks.rados.rados.0.trial039.stdout:401: snap_remove snap 48 2026-02-25T08:43:26.855 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:26.933 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:26.931+0000 7fc9dc6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:26.936 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:26.935+0000 7fc9dc6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:26.962 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:26.978 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:27.104 INFO:tasks.rados.rados.0.trial039.stdout:401: done (1 left) 2026-02-25T08:43:27.105 INFO:tasks.rados.rados.0.trial039.stdout:402: snap_create 2026-02-25T08:43:27.105 INFO:tasks.rados.rados.0.trial039.stdout:400: finishing write tid 1 to trial03942481-22 2026-02-25T08:43:27.105 INFO:tasks.rados.rados.0.trial039.stdout:400: finishing write tid 2 to trial03942481-22 2026-02-25T08:43:27.105 INFO:tasks.rados.rados.0.trial039.stdout:400: finishing write tid 3 to trial03942481-22 2026-02-25T08:43:27.105 INFO:tasks.rados.rados.0.trial039.stdout:400: finishing write tid 4 to trial03942481-22 2026-02-25T08:43:27.105 INFO:tasks.rados.rados.0.trial039.stdout:400: finishing write tid 5 to trial03942481-22 2026-02-25T08:43:27.105 INFO:tasks.rados.rados.0.trial039.stdout:400: oid 22 updating version 0 to 24 2026-02-25T08:43:27.105 INFO:tasks.rados.rados.0.trial039.stdout:400: oid 22 version 24 is already newer than 24 2026-02-25T08:43:27.105 INFO:tasks.rados.rados.0.trial039.stdout:400: oid 22 version 24 is already newer than 24 2026-02-25T08:43:27.105 INFO:tasks.rados.rados.0.trial039.stdout:400: oid 22 version 24 is already newer than 24 2026-02-25T08:43:27.106 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 22 v 24 (ObjNum 200 snap 51 seq_num 200) dirty exists 2026-02-25T08:43:27.106 INFO:tasks.rados.rados.0.trial039.stdout:400: left oid 22 (ObjNum 200 snap 51 seq_num 200) 2026-02-25T08:43:28.106 INFO:tasks.rados.rados.0.trial039.stdout:400: done (1 left) 2026-02-25T08:43:28.106 INFO:tasks.rados.rados.0.trial039.stdout:402: done (0 left) 2026-02-25T08:43:28.106 INFO:tasks.rados.rados.0.trial039.stdout:403: snap_create 2026-02-25T08:43:28.979 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:29.057 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:29.054+0000 7fac9e68b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:29.059 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:29.058+0000 7fac9e68b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:29.085 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:29.102 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:29.107 INFO:tasks.rados.rados.0.trial039.stdout:403: done (0 left) 2026-02-25T08:43:29.107 INFO:tasks.rados.rados.0.trial039.stdout:404: snap_remove snap 39 2026-02-25T08:43:30.112 INFO:tasks.rados.rados.0.trial039.stdout:404: done (0 left) 2026-02-25T08:43:30.112 INFO:tasks.rados.rados.0.trial039.stdout:405: write oid 46 current snap is 53 2026-02-25T08:43:30.112 INFO:tasks.rados.rados.0.trial039.stdout:405: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2026-02-25T08:43:30.116 INFO:tasks.rados.rados.0.trial039.stdout:405: writing trial03942481-46 from 578207 to 1128434 tid 1 2026-02-25T08:43:30.122 INFO:tasks.rados.rados.0.trial039.stdout:405: writing trial03942481-46 from 1762410 to 2542144 tid 2 2026-02-25T08:43:30.125 INFO:tasks.rados.rados.0.trial039.stdout:405: writing trial03942481-46 from 3286845 to 3378207 tid 3 2026-02-25T08:43:30.125 INFO:tasks.rados.rados.0.trial039.stdout:406: snap_remove snap 45 2026-02-25T08:43:30.517 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:30.589 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:30.592 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:30.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~116s 2026-02-25T08:43:31.104 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:31.117 INFO:tasks.rados.rados.0.trial039.stdout:406: done (1 left) 2026-02-25T08:43:31.117 INFO:tasks.rados.rados.0.trial039.stdout:407: delete oid 37 current snap is 53 2026-02-25T08:43:31.117 INFO:tasks.rados.rados.0.trial039.stdout:405: finishing write tid 1 to trial03942481-46 2026-02-25T08:43:31.117 INFO:tasks.rados.rados.0.trial039.stdout:405: finishing write tid 2 to trial03942481-46 2026-02-25T08:43:31.117 INFO:tasks.rados.rados.0.trial039.stdout:405: finishing write tid 3 to trial03942481-46 2026-02-25T08:43:31.117 INFO:tasks.rados.rados.0.trial039.stdout:405: finishing write tid 4 to trial03942481-46 2026-02-25T08:43:31.118 INFO:tasks.rados.rados.0.trial039.stdout:405: finishing write tid 5 to trial03942481-46 2026-02-25T08:43:31.118 INFO:tasks.rados.rados.0.trial039.stdout:405: oid 46 updating version 0 to 46 2026-02-25T08:43:31.118 INFO:tasks.rados.rados.0.trial039.stdout:405: oid 46 version 46 is already newer than 46 2026-02-25T08:43:31.118 INFO:tasks.rados.rados.0.trial039.stdout:405: oid 46 version 46 is already newer than 46 2026-02-25T08:43:31.118 INFO:tasks.rados.rados.0.trial039.stdout:405: oid 46 version 46 is already newer than 46 2026-02-25T08:43:31.118 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 46 v 46 (ObjNum 201 snap 53 seq_num 201) dirty exists 2026-02-25T08:43:31.118 INFO:tasks.rados.rados.0.trial039.stdout:405: left oid 46 (ObjNum 201 snap 53 seq_num 201) 2026-02-25T08:43:31.155 INFO:tasks.rados.rados.0.trial039.stdout:405: done (1 left) 2026-02-25T08:43:31.155 INFO:tasks.rados.rados.0.trial039.stdout:407: done (0 left) 2026-02-25T08:43:31.155 INFO:tasks.rados.rados.0.trial039.stdout:408: delete oid 48 current snap is 53 2026-02-25T08:43:31.158 INFO:tasks.rados.rados.0.trial039.stdout:408: done (0 left) 2026-02-25T08:43:31.158 INFO:tasks.rados.rados.0.trial039.stdout:409: read oid 9 snap -1 2026-02-25T08:43:31.158 INFO:tasks.rados.rados.0.trial039.stdout:409: expect deleted 2026-02-25T08:43:31.158 INFO:tasks.rados.rados.0.trial039.stdout:410: read oid 42 snap -1 2026-02-25T08:43:31.158 INFO:tasks.rados.rados.0.trial039.stdout:410: expect deleted 2026-02-25T08:43:31.158 INFO:tasks.rados.rados.0.trial039.stdout:411: write (excl) oid 4 current snap is 53 2026-02-25T08:43:31.158 INFO:tasks.rados.rados.0.trial039.stdout:411: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2026-02-25T08:43:31.163 INFO:tasks.rados.rados.0.trial039.stdout:411: writing trial03942481-4 from 628628 to 1404602 tid 1 2026-02-25T08:43:31.167 INFO:tasks.rados.rados.0.trial039.stdout:411: writing trial03942481-4 from 1937856 to 2464917 tid 2 2026-02-25T08:43:31.171 INFO:tasks.rados.rados.0.trial039.stdout:411: writing trial03942481-4 from 3252761 to 3428628 tid 3 2026-02-25T08:43:31.171 INFO:tasks.rados.rados.0.trial039.stdout:412: read oid 5 snap -1 2026-02-25T08:43:31.171 INFO:tasks.rados.rados.0.trial039.stdout:412: expect (ObjNum 152 snap 29 seq_num 152) 2026-02-25T08:43:31.171 INFO:tasks.rados.rados.0.trial039.stdout:413: snap_create 2026-02-25T08:43:31.184 INFO:tasks.rados.rados.0.trial039.stdout:412: expect (ObjNum 152 snap 29 seq_num 152) 2026-02-25T08:43:31.186 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:31.183+0000 7f0f01ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:31.189 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:31.188+0000 7f0f01ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:31.203 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:31.232 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:31.248 INFO:tasks.rados.rados.0.trial039.stdout:411: finishing write tid 1 to trial03942481-4 2026-02-25T08:43:31.248 INFO:tasks.rados.rados.0.trial039.stdout:411: finishing write tid 2 to trial03942481-4 2026-02-25T08:43:31.248 INFO:tasks.rados.rados.0.trial039.stdout:411: finishing write tid 3 to trial03942481-4 2026-02-25T08:43:31.248 INFO:tasks.rados.rados.0.trial039.stdout:411: finishing write tid 4 to trial03942481-4 2026-02-25T08:43:31.248 INFO:tasks.rados.rados.0.trial039.stdout:411: finishing write tid 5 to trial03942481-4 2026-02-25T08:43:31.248 INFO:tasks.rados.rados.0.trial039.stdout:411: oid 4 updating version 0 to 72 2026-02-25T08:43:31.248 INFO:tasks.rados.rados.0.trial039.stdout:411: oid 4 version 72 is already newer than 72 2026-02-25T08:43:31.248 INFO:tasks.rados.rados.0.trial039.stdout:411: oid 4 version 72 is already newer than 69 2026-02-25T08:43:31.248 INFO:tasks.rados.rados.0.trial039.stdout:411: oid 4 version 72 is already newer than 72 2026-02-25T08:43:31.248 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 72 (ObjNum 204 snap 53 seq_num 204) dirty exists 2026-02-25T08:43:31.249 INFO:tasks.rados.rados.0.trial039.stdout:411: left oid 4 (ObjNum 204 snap 53 seq_num 204) 2026-02-25T08:43:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:31.250 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:32.121 INFO:tasks.rados.rados.0.trial039.stdout:409: done (4 left) 2026-02-25T08:43:32.121 INFO:tasks.rados.rados.0.trial039.stdout:410: done (3 left) 2026-02-25T08:43:32.122 INFO:tasks.rados.rados.0.trial039.stdout:411: done (2 left) 2026-02-25T08:43:32.122 INFO:tasks.rados.rados.0.trial039.stdout:412: done (1 left) 2026-02-25T08:43:32.122 INFO:tasks.rados.rados.0.trial039.stdout:413: done (0 left) 2026-02-25T08:43:32.122 INFO:tasks.rados.rados.0.trial039.stdout:414: write (excl) oid 19 current snap is 54 2026-02-25T08:43:32.122 INFO:tasks.rados.rados.0.trial039.stdout:414: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2026-02-25T08:43:32.126 INFO:tasks.rados.rados.0.trial039.stdout:414: writing trial03942481-19 from 645435 to 1096658 tid 1 2026-02-25T08:43:32.133 INFO:tasks.rados.rados.0.trial039.stdout:414: writing trial03942481-19 from 1879985 to 2667057 tid 2 2026-02-25T08:43:32.135 INFO:tasks.rados.rados.0.trial039.stdout:414: writing trial03942481-19 from 3280184 to 3445435 tid 3 2026-02-25T08:43:32.136 INFO:tasks.rados.rados.0.trial039.stdout:415: snap_remove snap 50 2026-02-25T08:43:33.063 INFO:tasks.rados.rados.0.trial039.stdout:415: done (1 left) 2026-02-25T08:43:33.063 INFO:tasks.rados.rados.0.trial039.stdout:416: write (excl) oid 50 current snap is 54 2026-02-25T08:43:33.063 INFO:tasks.rados.rados.0.trial039.stdout:414: finishing write tid 1 to trial03942481-19 2026-02-25T08:43:33.063 INFO:tasks.rados.rados.0.trial039.stdout:416: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2026-02-25T08:43:33.067 INFO:tasks.rados.rados.0.trial039.stdout:416: writing trial03942481-50 from 662242 to 1305067 tid 1 2026-02-25T08:43:33.073 INFO:tasks.rados.rados.0.trial039.stdout:416: writing trial03942481-50 from 2054820 to 2585550 tid 2 2026-02-25T08:43:33.077 INFO:tasks.rados.rados.0.trial039.stdout:416: writing trial03942481-50 from 3307607 to 3462242 tid 3 2026-02-25T08:43:33.077 INFO:tasks.rados.rados.0.trial039.stdout:417: snap_create 2026-02-25T08:43:33.077 INFO:tasks.rados.rados.0.trial039.stdout:414: finishing write tid 2 to trial03942481-19 2026-02-25T08:43:33.077 INFO:tasks.rados.rados.0.trial039.stdout:414: finishing write tid 3 to trial03942481-19 2026-02-25T08:43:33.077 INFO:tasks.rados.rados.0.trial039.stdout:414: finishing write tid 4 to trial03942481-19 2026-02-25T08:43:33.077 INFO:tasks.rados.rados.0.trial039.stdout:414: finishing write tid 5 to trial03942481-19 2026-02-25T08:43:33.077 INFO:tasks.rados.rados.0.trial039.stdout:414: oid 19 updating version 0 to 127 2026-02-25T08:43:33.078 INFO:tasks.rados.rados.0.trial039.stdout:414: oid 19 version 127 is already newer than 127 2026-02-25T08:43:33.078 INFO:tasks.rados.rados.0.trial039.stdout:414: oid 19 version 127 is already newer than 127 2026-02-25T08:43:33.078 INFO:tasks.rados.rados.0.trial039.stdout:414: oid 19 version 127 is already newer than 122 2026-02-25T08:43:33.078 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 19 v 127 (ObjNum 205 snap 54 seq_num 205) dirty exists 2026-02-25T08:43:33.078 INFO:tasks.rados.rados.0.trial039.stdout:414: left oid 19 (ObjNum 205 snap 54 seq_num 205) 2026-02-25T08:43:33.114 INFO:tasks.rados.rados.0.trial039.stdout:416: finishing write tid 1 to trial03942481-50 2026-02-25T08:43:33.129 INFO:tasks.rados.rados.0.trial039.stdout:416: finishing write tid 2 to trial03942481-50 2026-02-25T08:43:33.135 INFO:tasks.rados.rados.0.trial039.stdout:416: finishing write tid 3 to trial03942481-50 2026-02-25T08:43:33.140 INFO:tasks.rados.rados.0.trial039.stdout:416: finishing write tid 4 to trial03942481-50 2026-02-25T08:43:33.141 INFO:tasks.rados.rados.0.trial039.stdout:416: finishing write tid 5 to trial03942481-50 2026-02-25T08:43:33.141 INFO:tasks.rados.rados.0.trial039.stdout:416: oid 50 updating version 0 to 92 2026-02-25T08:43:33.142 INFO:tasks.rados.rados.0.trial039.stdout:416: oid 50 version 92 is already newer than 89 2026-02-25T08:43:33.142 INFO:tasks.rados.rados.0.trial039.stdout:416: oid 50 version 92 is already newer than 92 2026-02-25T08:43:33.142 INFO:tasks.rados.rados.0.trial039.stdout:416: oid 50 version 92 is already newer than 92 2026-02-25T08:43:33.142 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 50 v 92 (ObjNum 206 snap 54 seq_num 206) dirty exists 2026-02-25T08:43:33.142 INFO:tasks.rados.rados.0.trial039.stdout:416: left oid 50 (ObjNum 206 snap 54 seq_num 206) 2026-02-25T08:43:33.251 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:33.329 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:33.326+0000 7f09dfe11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:33.331 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:33.330+0000 7f09dfe11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:33.358 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:33.375 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:34.064 INFO:tasks.rados.rados.0.trial039.stdout:414: done (2 left) 2026-02-25T08:43:34.065 INFO:tasks.rados.rados.0.trial039.stdout:416: done (1 left) 2026-02-25T08:43:34.065 INFO:tasks.rados.rados.0.trial039.stdout:417: done (0 left) 2026-02-25T08:43:34.065 INFO:tasks.rados.rados.0.trial039.stdout:418: write (excl) oid 2 current snap is 55 2026-02-25T08:43:34.065 INFO:tasks.rados.rados.0.trial039.stdout:418: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2026-02-25T08:43:34.069 INFO:tasks.rados.rados.0.trial039.stdout:418: writing trial03942481-2 from 679049 to 1397123 tid 1 2026-02-25T08:43:34.074 INFO:tasks.rados.rados.0.trial039.stdout:418: writing trial03942481-2 from 2113302 to 2787690 tid 2 2026-02-25T08:43:34.078 INFO:tasks.rados.rados.0.trial039.stdout:418: writing trial03942481-2 from 3335030 to 3479049 tid 3 2026-02-25T08:43:34.078 INFO:tasks.rados.rados.0.trial039.stdout:419: read oid 35 snap -1 2026-02-25T08:43:34.078 INFO:tasks.rados.rados.0.trial039.stdout:419: expect (ObjNum 139 snap 22 seq_num 139) 2026-02-25T08:43:34.078 INFO:tasks.rados.rados.0.trial039.stdout:420: read oid 7 snap -1 2026-02-25T08:43:34.078 INFO:tasks.rados.rados.0.trial039.stdout:420: expect deleted 2026-02-25T08:43:34.078 INFO:tasks.rados.rados.0.trial039.stdout:421: write (excl) oid 14 current snap is 55 2026-02-25T08:43:34.078 INFO:tasks.rados.rados.0.trial039.stdout:421: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2026-02-25T08:43:34.083 INFO:tasks.rados.rados.0.trial039.stdout:421: writing trial03942481-14 from 695856 to 1489179 tid 1 2026-02-25T08:43:34.087 INFO:tasks.rados.rados.0.trial039.stdout:421: writing trial03942481-14 from 2171784 to 2706183 tid 2 2026-02-25T08:43:34.091 INFO:tasks.rados.rados.0.trial039.stdout:421: writing trial03942481-14 from 3362453 to 3495856 tid 3 2026-02-25T08:43:34.091 INFO:tasks.rados.rados.0.trial039.stdout:422: read oid 16 snap 49 2026-02-25T08:43:34.091 INFO:tasks.rados.rados.0.trial039.stdout:422: expect (ObjNum 153 snap 29 seq_num 153) 2026-02-25T08:43:34.091 INFO:tasks.rados.rados.0.trial039.stdout:423: delete oid 19 current snap is 55 2026-02-25T08:43:34.091 INFO:tasks.rados.rados.0.trial039.stdout:419: expect (ObjNum 139 snap 22 seq_num 139) 2026-02-25T08:43:34.146 INFO:tasks.rados.rados.0.trial039.stdout:418: finishing write tid 1 to trial03942481-2 2026-02-25T08:43:34.147 INFO:tasks.rados.rados.0.trial039.stdout:421: finishing write tid 1 to trial03942481-14 2026-02-25T08:43:34.147 INFO:tasks.rados.rados.0.trial039.stdout:418: finishing write tid 2 to trial03942481-2 2026-02-25T08:43:34.147 INFO:tasks.rados.rados.0.trial039.stdout:422: expect (ObjNum 153 snap 29 seq_num 153) 2026-02-25T08:43:34.147 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:34.189 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:34.212 INFO:tasks.rados.rados.0.trial039.stdout:421: finishing write tid 2 to trial03942481-14 2026-02-25T08:43:34.212 INFO:tasks.rados.rados.0.trial039.stdout:418: finishing write tid 3 to trial03942481-2 2026-02-25T08:43:34.212 INFO:tasks.rados.rados.0.trial039.stdout:421: finishing write tid 3 to trial03942481-14 2026-02-25T08:43:34.212 INFO:tasks.rados.rados.0.trial039.stdout:418: finishing write tid 4 to trial03942481-2 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:419: done (5 left) 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:420: done (4 left) 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:422: done (3 left) 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:423: done (2 left) 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:424: write oid 12 current snap is 55 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:418: finishing write tid 5 to trial03942481-2 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:418: oid 2 updating version 0 to 77 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:418: oid 2 version 77 is already newer than 77 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:418: oid 2 version 77 is already newer than 77 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:418: oid 2 version 77 is already newer than 74 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 77 (ObjNum 207 snap 55 seq_num 207) dirty exists 2026-02-25T08:43:34.213 INFO:tasks.rados.rados.0.trial039.stdout:418: left oid 2 (ObjNum 207 snap 55 seq_num 207) 2026-02-25T08:43:34.214 INFO:tasks.rados.rados.0.trial039.stdout:421: finishing write tid 4 to trial03942481-14 2026-02-25T08:43:34.214 INFO:tasks.rados.rados.0.trial039.stdout:421: finishing write tid 5 to trial03942481-14 2026-02-25T08:43:34.214 INFO:tasks.rados.rados.0.trial039.stdout:421: oid 14 updating version 0 to 97 2026-02-25T08:43:34.214 INFO:tasks.rados.rados.0.trial039.stdout:421: oid 14 version 97 is already newer than 97 2026-02-25T08:43:34.214 INFO:tasks.rados.rados.0.trial039.stdout:421: oid 14 version 97 is already newer than 94 2026-02-25T08:43:34.214 INFO:tasks.rados.rados.0.trial039.stdout:421: oid 14 version 97 is already newer than 97 2026-02-25T08:43:34.214 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 14 v 97 (ObjNum 208 snap 55 seq_num 208) dirty exists 2026-02-25T08:43:34.214 INFO:tasks.rados.rados.0.trial039.stdout:421: left oid 14 (ObjNum 208 snap 55 seq_num 208) 2026-02-25T08:43:34.214 INFO:tasks.rados.rados.0.trial039.stdout:424: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2026-02-25T08:43:34.216 INFO:tasks.rados.rados.0.trial039.stdout:424: writing trial03942481-12 from 729470 to 1273291 tid 1 2026-02-25T08:43:34.220 INFO:tasks.rados.rados.0.trial039.stdout:424: writing trial03942481-12 from 1772395 to 2310463 tid 2 2026-02-25T08:43:34.224 INFO:tasks.rados.rados.0.trial039.stdout:424: writing trial03942481-12 from 2900946 to 3303951 tid 3 2026-02-25T08:43:34.224 INFO:tasks.rados.rados.0.trial039.stdout:418: done (2 left) 2026-02-25T08:43:34.224 INFO:tasks.rados.rados.0.trial039.stdout:421: done (1 left) 2026-02-25T08:43:34.224 INFO:tasks.rados.rados.0.trial039.stdout:425: write (excl) oid 2 current snap is 55 2026-02-25T08:43:34.224 INFO:tasks.rados.rados.0.trial039.stdout:425: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2026-02-25T08:43:34.229 INFO:tasks.rados.rados.0.trial039.stdout:425: writing trial03942481-2 from 746277 to 1365347 tid 1 2026-02-25T08:43:34.234 INFO:tasks.rados.rados.0.trial039.stdout:425: writing trial03942481-2 from 1830877 to 2512603 tid 2 2026-02-25T08:43:34.238 INFO:tasks.rados.rados.0.trial039.stdout:425: writing trial03942481-2 from 2928369 to 3546277 tid 3 2026-02-25T08:43:34.238 INFO:tasks.rados.rados.0.trial039.stdout:426: write (excl) oid 31 current snap is 55 2026-02-25T08:43:34.238 INFO:tasks.rados.rados.0.trial039.stdout:424: finishing write tid 1 to trial03942481-12 2026-02-25T08:43:34.238 INFO:tasks.rados.rados.0.trial039.stdout:426: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2026-02-25T08:43:34.243 INFO:tasks.rados.rados.0.trial039.stdout:426: writing trial03942481-31 from 763084 to 1457403 tid 1 2026-02-25T08:43:34.247 INFO:tasks.rados.rados.0.trial039.stdout:426: writing trial03942481-31 from 1889359 to 2431096 tid 2 2026-02-25T08:43:34.252 INFO:tasks.rados.rados.0.trial039.stdout:426: writing trial03942481-31 from 2955792 to 3497694 tid 3 2026-02-25T08:43:34.252 INFO:tasks.rados.rados.0.trial039.stdout:427: read oid 23 snap -1 2026-02-25T08:43:34.252 INFO:tasks.rados.rados.0.trial039.stdout:427: expect deleted 2026-02-25T08:43:34.252 INFO:tasks.rados.rados.0.trial039.stdout:428: read oid 1 snap 47 2026-02-25T08:43:34.252 INFO:tasks.rados.rados.0.trial039.stdout:428: expect (ObjNum 102 snap 11 seq_num 102) 2026-02-25T08:43:34.252 INFO:tasks.rados.rados.0.trial039.stdout:429: delete oid 21 current snap is 55 2026-02-25T08:43:34.253 INFO:tasks.rados.rados.0.trial039.stdout:424: finishing write tid 2 to trial03942481-12 2026-02-25T08:43:34.253 INFO:tasks.rados.rados.0.trial039.stdout:425: finishing write tid 1 to trial03942481-2 2026-02-25T08:43:34.253 INFO:tasks.rados.rados.0.trial039.stdout:424: finishing write tid 3 to trial03942481-12 2026-02-25T08:43:34.267 INFO:tasks.rados.rados.0.trial039.stdout:424: finishing write tid 4 to trial03942481-12 2026-02-25T08:43:34.271 INFO:tasks.rados.rados.0.trial039.stdout:424: finishing write tid 5 to trial03942481-12 2026-02-25T08:43:34.271 INFO:tasks.rados.rados.0.trial039.stdout:424: oid 12 updating version 0 to 33 2026-02-25T08:43:34.271 INFO:tasks.rados.rados.0.trial039.stdout:424: oid 12 version 33 is already newer than 33 2026-02-25T08:43:34.271 INFO:tasks.rados.rados.0.trial039.stdout:424: oid 12 version 33 is already newer than 33 2026-02-25T08:43:34.271 INFO:tasks.rados.rados.0.trial039.stdout:424: oid 12 version 33 is already newer than 33 2026-02-25T08:43:34.271 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 12 v 33 (ObjNum 210 snap 55 seq_num 210) dirty exists 2026-02-25T08:43:34.271 INFO:tasks.rados.rados.0.trial039.stdout:424: left oid 12 (ObjNum 210 snap 55 seq_num 210) 2026-02-25T08:43:34.279 INFO:tasks.rados.rados.0.trial039.stdout:428: expect (ObjNum 102 snap 11 seq_num 102) 2026-02-25T08:43:34.287 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1661121 2026-02-25T08:43:34.323 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1660858 2026-02-25T08:43:34.394 INFO:tasks.rados.rados.0.trial039.stdout:424: done (5 left) 2026-02-25T08:43:34.394 INFO:tasks.rados.rados.0.trial039.stdout:427: done (4 left) 2026-02-25T08:43:34.394 INFO:tasks.rados.rados.0.trial039.stdout:428: done (3 left) 2026-02-25T08:43:34.394 INFO:tasks.rados.rados.0.trial039.stdout:429: done (2 left) 2026-02-25T08:43:34.394 INFO:tasks.rados.rados.0.trial039.stdout:430: write oid 22 current snap is 55 2026-02-25T08:43:34.394 INFO:tasks.rados.rados.0.trial039.stdout:425: finishing write tid 2 to trial03942481-2 2026-02-25T08:43:34.394 INFO:tasks.rados.rados.0.trial039.stdout:430: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2026-02-25T08:43:34.398 INFO:tasks.rados.rados.0.trial039.stdout:430: writing trial03942481-22 from 796698 to 1357868 tid 1 2026-02-25T08:43:34.403 INFO:tasks.rados.rados.0.trial039.stdout:430: writing trial03942481-22 from 2006323 to 2551729 tid 2 2026-02-25T08:43:34.407 INFO:tasks.rados.rados.0.trial039.stdout:430: writing trial03942481-22 from 3126991 to 3596698 tid 3 2026-02-25T08:43:34.407 INFO:tasks.rados.rados.0.trial039.stdout:431: snap_create 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:425: finishing write tid 3 to trial03942481-2 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:425: finishing write tid 4 to trial03942481-2 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:425: finishing write tid 5 to trial03942481-2 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:425: oid 2 updating version 0 to 81 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:425: oid 2 version 81 is already newer than 81 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:425: oid 2 version 81 is already newer than 81 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:425: oid 2 version 81 is already newer than 78 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 81 (ObjNum 211 snap 55 seq_num 211) dirty exists 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:425: left oid 2 (ObjNum 211 snap 55 seq_num 211) 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:426: finishing write tid 1 to trial03942481-31 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:426: finishing write tid 2 to trial03942481-31 2026-02-25T08:43:34.408 INFO:tasks.rados.rados.0.trial039.stdout:426: finishing write tid 3 to trial03942481-31 2026-02-25T08:43:34.409 INFO:tasks.rados.rados.0.trial039.stdout:426: finishing write tid 4 to trial03942481-31 2026-02-25T08:43:34.409 INFO:tasks.rados.rados.0.trial039.stdout:426: finishing write tid 5 to trial03942481-31 2026-02-25T08:43:34.409 INFO:tasks.rados.rados.0.trial039.stdout:426: oid 31 updating version 0 to 136 2026-02-25T08:43:34.409 INFO:tasks.rados.rados.0.trial039.stdout:426: oid 31 version 136 is already newer than 133 2026-02-25T08:43:34.409 INFO:tasks.rados.rados.0.trial039.stdout:426: oid 31 version 136 is already newer than 136 2026-02-25T08:43:34.409 INFO:tasks.rados.rados.0.trial039.stdout:426: oid 31 version 136 is already newer than 136 2026-02-25T08:43:34.409 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 136 (ObjNum 212 snap 55 seq_num 212) dirty exists 2026-02-25T08:43:34.409 INFO:tasks.rados.rados.0.trial039.stdout:426: left oid 31 (ObjNum 212 snap 55 seq_num 212) 2026-02-25T08:43:34.430 INFO:tasks.rados.rados.0.trial039.stdout:430: finishing write tid 1 to trial03942481-22 2026-02-25T08:43:34.437 INFO:tasks.rados.rados.0.trial039.stdout:430: finishing write tid 2 to trial03942481-22 2026-02-25T08:43:34.445 INFO:tasks.rados.rados.0.trial039.stdout:430: finishing write tid 3 to trial03942481-22 2026-02-25T08:43:34.447 INFO:tasks.rados.rados.0.trial039.stdout:430: finishing write tid 4 to trial03942481-22 2026-02-25T08:43:34.449 INFO:tasks.rados.rados.0.trial039.stdout:430: finishing write tid 5 to trial03942481-22 2026-02-25T08:43:34.449 INFO:tasks.rados.rados.0.trial039.stdout:430: oid 22 updating version 0 to 29 2026-02-25T08:43:34.449 INFO:tasks.rados.rados.0.trial039.stdout:430: oid 22 version 29 is already newer than 29 2026-02-25T08:43:34.449 INFO:tasks.rados.rados.0.trial039.stdout:430: oid 22 version 29 is already newer than 29 2026-02-25T08:43:34.449 INFO:tasks.rados.rados.0.trial039.stdout:430: oid 22 version 29 is already newer than 29 2026-02-25T08:43:34.449 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 22 v 29 (ObjNum 214 snap 55 seq_num 214) dirty exists 2026-02-25T08:43:34.449 INFO:tasks.rados.rados.0.trial039.stdout:430: left oid 22 (ObjNum 214 snap 55 seq_num 214) 2026-02-25T08:43:35.129 INFO:tasks.rados.rados.0.trial039.stdout:425: done (3 left) 2026-02-25T08:43:35.129 INFO:tasks.rados.rados.0.trial039.stdout:426: done (2 left) 2026-02-25T08:43:35.129 INFO:tasks.rados.rados.0.trial039.stdout:430: done (1 left) 2026-02-25T08:43:35.129 INFO:tasks.rados.rados.0.trial039.stdout:431: done (0 left) 2026-02-25T08:43:35.129 INFO:tasks.rados.rados.0.trial039.stdout:432: write oid 8 current snap is 56 2026-02-25T08:43:35.129 INFO:tasks.rados.rados.0.trial039.stdout:432: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2026-02-25T08:43:35.132 INFO:tasks.rados.rados.0.trial039.stdout:432: writing trial03942481-8 from 413505 to 1049924 tid 1 2026-02-25T08:43:35.138 INFO:tasks.rados.rados.0.trial039.stdout:432: writing trial03942481-8 from 1664805 to 2353869 tid 2 2026-02-25T08:43:35.143 INFO:tasks.rados.rados.0.trial039.stdout:432: writing trial03942481-8 from 3038061 to 3530132 tid 3 2026-02-25T08:43:35.143 INFO:tasks.rados.rados.0.trial039.stdout:433: read oid 30 snap -1 2026-02-25T08:43:35.143 INFO:tasks.rados.rados.0.trial039.stdout:433: expect (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:43:35.143 INFO:tasks.rados.rados.0.trial039.stdout:434: read oid 34 snap 47 2026-02-25T08:43:35.143 INFO:tasks.rados.rados.0.trial039.stdout:434: expect (ObjNum 151 snap 29 seq_num 151) 2026-02-25T08:43:35.144 INFO:tasks.rados.rados.0.trial039.stdout:435: write oid 44 current snap is 56 2026-02-25T08:43:35.144 INFO:tasks.rados.rados.0.trial039.stdout:435: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2026-02-25T08:43:35.147 INFO:tasks.rados.rados.0.trial039.stdout:435: writing trial03942481-44 from 430312 to 1141980 tid 1 2026-02-25T08:43:35.151 INFO:tasks.rados.rados.0.trial039.stdout:435: writing trial03942481-44 from 1723287 to 2272362 tid 2 2026-02-25T08:43:35.156 INFO:tasks.rados.rados.0.trial039.stdout:435: writing trial03942481-44 from 2781837 to 3485180 tid 3 2026-02-25T08:43:35.156 INFO:tasks.rados.rados.0.trial039.stdout:436: read oid 45 snap -1 2026-02-25T08:43:35.156 INFO:tasks.rados.rados.0.trial039.stdout:436: expect (ObjNum 180 snap 40 seq_num 180) 2026-02-25T08:43:35.156 INFO:tasks.rados.rados.0.trial039.stdout:437: read oid 47 snap -1 2026-02-25T08:43:35.157 INFO:tasks.rados.rados.0.trial039.stdout:437: expect (ObjNum 80 snap 8 seq_num 80) 2026-02-25T08:43:35.157 INFO:tasks.rados.rados.0.trial039.stdout:438: write (excl) oid 17 current snap is 56 2026-02-25T08:43:35.157 INFO:tasks.rados.rados.0.trial039.stdout:438: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2026-02-25T08:43:35.161 INFO:tasks.rados.rados.0.trial039.stdout:438: writing trial03942481-17 from 447119 to 1234036 tid 1 2026-02-25T08:43:35.165 INFO:tasks.rados.rados.0.trial039.stdout:438: writing trial03942481-17 from 1665416 to 2358149 tid 2 2026-02-25T08:43:35.169 INFO:tasks.rados.rados.0.trial039.stdout:438: writing trial03942481-17 from 2976554 to 3491169 tid 3 2026-02-25T08:43:35.169 INFO:tasks.rados.rados.0.trial039.stdout:439: snap_create 2026-02-25T08:43:35.170 INFO:tasks.rados.rados.0.trial039.stdout:434: expect (ObjNum 151 snap 29 seq_num 151) 2026-02-25T08:43:35.272 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2518684 2026-02-25T08:43:35.274 INFO:tasks.rados.rados.0.trial039.stdout:437: expect (ObjNum 80 snap 8 seq_num 80) 2026-02-25T08:43:35.274 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:35.346 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:35.375 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:35.385 INFO:tasks.rados.rados.0.trial039.stdout:433: expect (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:43:35.437 INFO:tasks.rados.rados.0.trial039.stdout:432: finishing write tid 1 to trial03942481-8 2026-02-25T08:43:35.437 INFO:tasks.rados.rados.0.trial039.stdout:436: expect (ObjNum 180 snap 40 seq_num 180) 2026-02-25T08:43:35.445 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1057578 2026-02-25T08:43:35.451 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:35.449+0000 7fde31b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:35.454 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:35.453+0000 7fde31b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:35.481 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:35.498 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:35.530 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2767582 2026-02-25T08:43:35.560 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1861088 2026-02-25T08:43:35.586 INFO:tasks.rados.rados.0.trial039.stdout:432: finishing write tid 2 to trial03942481-8 2026-02-25T08:43:35.586 INFO:tasks.rados.rados.0.trial039.stdout:432: finishing write tid 3 to trial03942481-8 2026-02-25T08:43:35.586 INFO:tasks.rados.rados.0.trial039.stdout:432: finishing write tid 4 to trial03942481-8 2026-02-25T08:43:35.586 INFO:tasks.rados.rados.0.trial039.stdout:432: finishing write tid 5 to trial03942481-8 2026-02-25T08:43:35.586 INFO:tasks.rados.rados.0.trial039.stdout:432: oid 8 updating version 0 to 70 2026-02-25T08:43:35.586 INFO:tasks.rados.rados.0.trial039.stdout:432: oid 8 version 70 is already newer than 70 2026-02-25T08:43:35.586 INFO:tasks.rados.rados.0.trial039.stdout:432: oid 8 version 70 is already newer than 70 2026-02-25T08:43:35.586 INFO:tasks.rados.rados.0.trial039.stdout:432: oid 8 version 70 is already newer than 70 2026-02-25T08:43:35.586 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 8 v 70 (ObjNum 215 snap 56 seq_num 215) dirty exists 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:432: left oid 8 (ObjNum 215 snap 56 seq_num 215) 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:435: finishing write tid 1 to trial03942481-44 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:435: finishing write tid 2 to trial03942481-44 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:435: finishing write tid 3 to trial03942481-44 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:435: finishing write tid 4 to trial03942481-44 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:435: finishing write tid 5 to trial03942481-44 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:435: oid 44 updating version 0 to 86 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:435: oid 44 version 86 is already newer than 86 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:435: oid 44 version 86 is already newer than 86 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:435: oid 44 version 86 is already newer than 86 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 86 (ObjNum 216 snap 56 seq_num 216) dirty exists 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:435: left oid 44 (ObjNum 216 snap 56 seq_num 216) 2026-02-25T08:43:35.587 INFO:tasks.rados.rados.0.trial039.stdout:438: finishing write tid 1 to trial03942481-17 2026-02-25T08:43:35.588 INFO:tasks.rados.rados.0.trial039.stdout:438: finishing write tid 2 to trial03942481-17 2026-02-25T08:43:35.588 INFO:tasks.rados.rados.0.trial039.stdout:438: finishing write tid 3 to trial03942481-17 2026-02-25T08:43:35.588 INFO:tasks.rados.rados.0.trial039.stdout:438: finishing write tid 4 to trial03942481-17 2026-02-25T08:43:35.588 INFO:tasks.rados.rados.0.trial039.stdout:438: finishing write tid 5 to trial03942481-17 2026-02-25T08:43:35.588 INFO:tasks.rados.rados.0.trial039.stdout:438: oid 17 updating version 0 to 91 2026-02-25T08:43:35.588 INFO:tasks.rados.rados.0.trial039.stdout:438: oid 17 version 91 is already newer than 91 2026-02-25T08:43:35.588 INFO:tasks.rados.rados.0.trial039.stdout:438: oid 17 version 91 is already newer than 88 2026-02-25T08:43:35.588 INFO:tasks.rados.rados.0.trial039.stdout:438: oid 17 version 91 is already newer than 91 2026-02-25T08:43:35.588 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 17 v 91 (ObjNum 217 snap 56 seq_num 217) dirty exists 2026-02-25T08:43:35.588 INFO:tasks.rados.rados.0.trial039.stdout:438: left oid 17 (ObjNum 217 snap 56 seq_num 217) 2026-02-25T08:43:35.593 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:35.665 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:35.669 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:36.129 INFO:tasks.rados.rados.0.trial039.stdout:432: done (7 left) 2026-02-25T08:43:36.129 INFO:tasks.rados.rados.0.trial039.stdout:433: done (6 left) 2026-02-25T08:43:36.130 INFO:tasks.rados.rados.0.trial039.stdout:434: done (5 left) 2026-02-25T08:43:36.130 INFO:tasks.rados.rados.0.trial039.stdout:435: done (4 left) 2026-02-25T08:43:36.130 INFO:tasks.rados.rados.0.trial039.stdout:436: done (3 left) 2026-02-25T08:43:36.130 INFO:tasks.rados.rados.0.trial039.stdout:437: done (2 left) 2026-02-25T08:43:36.130 INFO:tasks.rados.rados.0.trial039.stdout:438: done (1 left) 2026-02-25T08:43:36.130 INFO:tasks.rados.rados.0.trial039.stdout:439: done (0 left) 2026-02-25T08:43:36.130 INFO:tasks.rados.rados.0.trial039.stdout:440: delete oid 4 current snap is 57 2026-02-25T08:43:36.169 INFO:tasks.rados.rados.0.trial039.stdout:440: done (0 left) 2026-02-25T08:43:36.169 INFO:tasks.rados.rados.0.trial039.stdout:441: delete oid 1 current snap is 57 2026-02-25T08:43:36.198 INFO:tasks.rados.rados.0.trial039.stdout:441: done (0 left) 2026-02-25T08:43:36.198 INFO:tasks.rados.rados.0.trial039.stdout:442: read oid 41 snap 47 2026-02-25T08:43:36.198 INFO:tasks.rados.rados.0.trial039.stdout:442: expect (ObjNum 65 snap 4 seq_num 65) 2026-02-25T08:43:36.198 INFO:tasks.rados.rados.0.trial039.stdout:443: write oid 3 current snap is 57 2026-02-25T08:43:36.198 INFO:tasks.rados.rados.0.trial039.stdout:443: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2026-02-25T08:43:36.202 INFO:tasks.rados.rados.0.trial039.stdout:443: writing trial03942481-3 from 497540 to 1110204 tid 1 2026-02-25T08:43:36.206 INFO:tasks.rados.rados.0.trial039.stdout:443: writing trial03942481-3 from 1840862 to 2280922 tid 2 2026-02-25T08:43:36.211 INFO:tasks.rados.rados.0.trial039.stdout:443: writing trial03942481-3 from 3058823 to 3523607 tid 3 2026-02-25T08:43:36.211 INFO:tasks.rados.rados.0.trial039.stdout:444: write oid 47 current snap is 57 2026-02-25T08:43:36.212 INFO:tasks.rados.rados.0.trial039.stdout:444: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2026-02-25T08:43:36.214 INFO:tasks.rados.rados.0.trial039.stdout:444: writing trial03942481-47 from 514347 to 918613 tid 1 2026-02-25T08:43:36.219 INFO:tasks.rados.rados.0.trial039.stdout:444: writing trial03942481-47 from 1499344 to 2199415 tid 2 2026-02-25T08:43:36.224 INFO:tasks.rados.rados.0.trial039.stdout:444: writing trial03942481-47 from 2686246 to 3362302 tid 3 2026-02-25T08:43:36.224 INFO:tasks.rados.rados.0.trial039.stdout:445: snap_create 2026-02-25T08:43:36.224 INFO:tasks.rados.rados.0.trial039.stdout:442: expect (ObjNum 65 snap 4 seq_num 65) 2026-02-25T08:43:36.268 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2061295 2026-02-25T08:43:36.324 INFO:tasks.rados.rados.0.trial039.stdout:443: finishing write tid 1 to trial03942481-3 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:443: finishing write tid 2 to trial03942481-3 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:444: finishing write tid 1 to trial03942481-47 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:443: finishing write tid 3 to trial03942481-3 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:443: finishing write tid 4 to trial03942481-3 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:444: finishing write tid 2 to trial03942481-47 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:443: finishing write tid 5 to trial03942481-3 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:443: oid 3 updating version 0 to 143 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:443: oid 3 version 143 is already newer than 143 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:443: oid 3 version 143 is already newer than 143 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:443: oid 3 version 143 is already newer than 143 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 3 v 143 (ObjNum 220 snap 57 seq_num 220) dirty exists 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:443: left oid 3 (ObjNum 220 snap 57 seq_num 220) 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:444: finishing write tid 3 to trial03942481-47 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:444: finishing write tid 4 to trial03942481-47 2026-02-25T08:43:36.325 INFO:tasks.rados.rados.0.trial039.stdout:444: finishing write tid 5 to trial03942481-47 2026-02-25T08:43:36.326 INFO:tasks.rados.rados.0.trial039.stdout:444: oid 47 updating version 0 to 36 2026-02-25T08:43:36.326 INFO:tasks.rados.rados.0.trial039.stdout:444: oid 47 version 36 is already newer than 36 2026-02-25T08:43:36.326 INFO:tasks.rados.rados.0.trial039.stdout:444: oid 47 version 36 is already newer than 36 2026-02-25T08:43:36.326 INFO:tasks.rados.rados.0.trial039.stdout:444: oid 47 version 36 is already newer than 36 2026-02-25T08:43:36.326 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 47 v 36 (ObjNum 221 snap 57 seq_num 221) dirty exists 2026-02-25T08:43:36.326 INFO:tasks.rados.rados.0.trial039.stdout:444: left oid 47 (ObjNum 221 snap 57 seq_num 221) 2026-02-25T08:43:36.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~121s 2026-02-25T08:43:37.134 INFO:tasks.rados.rados.0.trial039.stdout:442: done (3 left) 2026-02-25T08:43:37.134 INFO:tasks.rados.rados.0.trial039.stdout:443: done (2 left) 2026-02-25T08:43:37.134 INFO:tasks.rados.rados.0.trial039.stdout:444: done (1 left) 2026-02-25T08:43:37.134 INFO:tasks.rados.rados.0.trial039.stdout:445: done (0 left) 2026-02-25T08:43:37.134 INFO:tasks.rados.rados.0.trial039.stdout:446: delete oid 13 current snap is 58 2026-02-25T08:43:37.148 INFO:tasks.rados.rados.0.trial039.stdout:446: done (0 left) 2026-02-25T08:43:37.148 INFO:tasks.rados.rados.0.trial039.stdout:447: delete oid 48 current snap is 58 2026-02-25T08:43:37.151 INFO:tasks.rados.rados.0.trial039.stdout:447: done (0 left) 2026-02-25T08:43:37.151 INFO:tasks.rados.rados.0.trial039.stdout:448: read oid 40 snap -1 2026-02-25T08:43:37.151 INFO:tasks.rados.rados.0.trial039.stdout:448: expect deleted 2026-02-25T08:43:37.151 INFO:tasks.rados.rados.0.trial039.stdout:449: snap_create 2026-02-25T08:43:37.500 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:37.575 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:37.573+0000 7ff57e7ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:37.578 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:37.577+0000 7ff57e7ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:37.603 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:37.619 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:37.619 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:38.137 INFO:tasks.rados.rados.0.trial039.stdout:448: done (1 left) 2026-02-25T08:43:38.138 INFO:tasks.rados.rados.0.trial039.stdout:449: done (0 left) 2026-02-25T08:43:38.138 INFO:tasks.rados.rados.0.trial039.stdout:450: snap_create 2026-02-25T08:43:39.141 INFO:tasks.rados.rados.0.trial039.stdout:450: done (0 left) 2026-02-25T08:43:39.141 INFO:tasks.rados.rados.0.trial039.stdout:451: read oid 32 snap -1 2026-02-25T08:43:39.142 INFO:tasks.rados.rados.0.trial039.stdout:451: expect (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:43:39.142 INFO:tasks.rados.rados.0.trial039.stdout:452: write (excl) oid 4 current snap is 60 2026-02-25T08:43:39.142 INFO:tasks.rados.rados.0.trial039.stdout:452: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2026-02-25T08:43:39.146 INFO:tasks.rados.rados.0.trial039.stdout:452: writing trial03942481-4 from 564768 to 1194781 tid 1 2026-02-25T08:43:39.151 INFO:tasks.rados.rados.0.trial039.stdout:452: writing trial03942481-4 from 1674790 to 2122188 tid 2 2026-02-25T08:43:39.156 INFO:tasks.rados.rados.0.trial039.stdout:452: writing trial03942481-4 from 2768515 to 3394740 tid 3 2026-02-25T08:43:39.156 INFO:tasks.rados.rados.0.trial039.stdout:453: delete oid 29 current snap is 60 2026-02-25T08:43:39.159 INFO:tasks.rados.rados.0.trial039.stdout:453: done (2 left) 2026-02-25T08:43:39.159 INFO:tasks.rados.rados.0.trial039.stdout:454: read oid 48 snap 52 2026-02-25T08:43:39.159 INFO:tasks.rados.rados.0.trial039.stdout:454: expect deleted 2026-02-25T08:43:39.160 INFO:tasks.rados.rados.0.trial039.stdout:455: snap_create 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial039.stdout:452: finishing write tid 1 to trial03942481-4 2026-02-25T08:43:39.172 INFO:tasks.rados.rados.0.trial039.stdout:451: expect (ObjNum 128 snap 20 seq_num 128) 2026-02-25T08:43:39.206 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 943736 2026-02-25T08:43:39.217 INFO:tasks.rados.rados.0.trial039.stdout:452: finishing write tid 2 to trial03942481-4 2026-02-25T08:43:39.217 INFO:tasks.rados.rados.0.trial039.stdout:452: finishing write tid 3 to trial03942481-4 2026-02-25T08:43:39.217 INFO:tasks.rados.rados.0.trial039.stdout:452: finishing write tid 4 to trial03942481-4 2026-02-25T08:43:39.217 INFO:tasks.rados.rados.0.trial039.stdout:452: finishing write tid 5 to trial03942481-4 2026-02-25T08:43:39.217 INFO:tasks.rados.rados.0.trial039.stdout:452: oid 4 updating version 0 to 75 2026-02-25T08:43:39.217 INFO:tasks.rados.rados.0.trial039.stdout:452: oid 4 updating version 75 to 78 2026-02-25T08:43:39.217 INFO:tasks.rados.rados.0.trial039.stdout:452: oid 4 version 78 is already newer than 78 2026-02-25T08:43:39.217 INFO:tasks.rados.rados.0.trial039.stdout:452: oid 4 version 78 is already newer than 78 2026-02-25T08:43:39.217 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 78 (ObjNum 224 snap 60 seq_num 224) dirty exists 2026-02-25T08:43:39.217 INFO:tasks.rados.rados.0.trial039.stdout:452: left oid 4 (ObjNum 224 snap 60 seq_num 224) 2026-02-25T08:43:39.620 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:39.695 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:39.693+0000 7f9657a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:39.697 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:39.696+0000 7f9657a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:39.723 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:39.739 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:40.145 INFO:tasks.rados.rados.0.trial039.stdout:451: done (3 left) 2026-02-25T08:43:40.146 INFO:tasks.rados.rados.0.trial039.stdout:452: done (2 left) 2026-02-25T08:43:40.146 INFO:tasks.rados.rados.0.trial039.stdout:454: done (1 left) 2026-02-25T08:43:40.146 INFO:tasks.rados.rados.0.trial039.stdout:455: done (0 left) 2026-02-25T08:43:40.146 INFO:tasks.rados.rados.0.trial039.stdout:456: snap_remove snap 55 2026-02-25T08:43:40.670 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:40.741 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:40.743 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:40.744 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:41.150 INFO:tasks.rados.rados.0.trial039.stdout:456: done (0 left) 2026-02-25T08:43:41.150 INFO:tasks.rados.rados.0.trial039.stdout:457: read oid 39 snap -1 2026-02-25T08:43:41.150 INFO:tasks.rados.rados.0.trial039.stdout:457: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:43:41.150 INFO:tasks.rados.rados.0.trial039.stdout:458: snap_remove snap 56 2026-02-25T08:43:41.741 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:41.819 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:41.817+0000 7f75f1f83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:41.822 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:41.821+0000 7f75f1f83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:41.847 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:41.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:41.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:41.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~127s 2026-02-25T08:43:42.152 INFO:tasks.rados.rados.0.trial039.stdout:458: done (1 left) 2026-02-25T08:43:42.152 INFO:tasks.rados.rados.0.trial039.stdout:459: write oid 34 current snap is 61 2026-02-25T08:43:42.152 INFO:tasks.rados.rados.0.trial039.stdout:459: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2026-02-25T08:43:42.157 INFO:tasks.rados.rados.0.trial039.stdout:459: writing trial03942481-34 from 598382 to 1378893 tid 1 2026-02-25T08:43:42.162 INFO:tasks.rados.rados.0.trial039.stdout:459: writing trial03942481-34 from 2075401 to 2526468 tid 2 2026-02-25T08:43:42.167 INFO:tasks.rados.rados.0.trial039.stdout:459: writing trial03942481-34 from 3107008 to 3755777 tid 3 2026-02-25T08:43:42.167 INFO:tasks.rados.rados.0.trial039.stdout:460: delete oid 33 current snap is 61 2026-02-25T08:43:42.167 INFO:tasks.rados.rados.0.trial039.stdout:457: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:43:42.198 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2572656 2026-02-25T08:43:42.225 INFO:tasks.rados.rados.0.trial039.stdout:459: finishing write tid 1 to trial03942481-34 2026-02-25T08:43:42.225 INFO:tasks.rados.rados.0.trial039.stdout:459: finishing write tid 2 to trial03942481-34 2026-02-25T08:43:42.225 INFO:tasks.rados.rados.0.trial039.stdout:459: finishing write tid 3 to trial03942481-34 2026-02-25T08:43:42.225 INFO:tasks.rados.rados.0.trial039.stdout:457: done (2 left) 2026-02-25T08:43:42.225 INFO:tasks.rados.rados.0.trial039.stdout:460: done (1 left) 2026-02-25T08:43:42.225 INFO:tasks.rados.rados.0.trial039.stdout:461: read oid 17 snap -1 2026-02-25T08:43:42.225 INFO:tasks.rados.rados.0.trial039.stdout:461: expect (ObjNum 217 snap 56 seq_num 217) 2026-02-25T08:43:42.225 INFO:tasks.rados.rados.0.trial039.stdout:462: read oid 10 snap -1 2026-02-25T08:43:42.225 INFO:tasks.rados.rados.0.trial039.stdout:462: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-25T08:43:42.225 INFO:tasks.rados.rados.0.trial039.stdout:463: snap_remove snap 58 2026-02-25T08:43:43.065 INFO:tasks.rados.rados.0.trial039.stdout:463: done (3 left) 2026-02-25T08:43:43.065 INFO:tasks.rados.rados.0.trial039.stdout:464: read oid 31 snap -1 2026-02-25T08:43:43.065 INFO:tasks.rados.rados.0.trial039.stdout:464: expect (ObjNum 212 snap 55 seq_num 212) 2026-02-25T08:43:43.065 INFO:tasks.rados.rados.0.trial039.stdout:465: snap_create 2026-02-25T08:43:43.065 INFO:tasks.rados.rados.0.trial039.stdout:459: finishing write tid 4 to trial03942481-34 2026-02-25T08:43:43.066 INFO:tasks.rados.rados.0.trial039.stdout:459: finishing write tid 5 to trial03942481-34 2026-02-25T08:43:43.066 INFO:tasks.rados.rados.0.trial039.stdout:459: oid 34 updating version 0 to 102 2026-02-25T08:43:43.066 INFO:tasks.rados.rados.0.trial039.stdout:459: oid 34 version 102 is already newer than 102 2026-02-25T08:43:43.066 INFO:tasks.rados.rados.0.trial039.stdout:459: oid 34 version 102 is already newer than 102 2026-02-25T08:43:43.066 INFO:tasks.rados.rados.0.trial039.stdout:459: oid 34 version 102 is already newer than 102 2026-02-25T08:43:43.066 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 102 (ObjNum 226 snap 61 seq_num 226) dirty exists 2026-02-25T08:43:43.066 INFO:tasks.rados.rados.0.trial039.stdout:459: left oid 34 (ObjNum 226 snap 61 seq_num 226) 2026-02-25T08:43:43.066 INFO:tasks.rados.rados.0.trial039.stdout:461: expect (ObjNum 217 snap 56 seq_num 217) 2026-02-25T08:43:43.096 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 3139876 2026-02-25T08:43:43.166 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1526697 2026-02-25T08:43:43.199 INFO:tasks.rados.rados.0.trial039.stdout:462: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-25T08:43:43.210 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1726806 2026-02-25T08:43:43.227 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:43.291 INFO:tasks.rados.rados.0.trial039.stdout:464: expect (ObjNum 212 snap 55 seq_num 212) 2026-02-25T08:43:43.375 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:43.865 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:43.943 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:43.940+0000 7f9141a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:43.945 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:43.944+0000 7f9141a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:43.970 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:43.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:44.066 INFO:tasks.rados.rados.0.trial039.stdout:459: done (4 left) 2026-02-25T08:43:44.066 INFO:tasks.rados.rados.0.trial039.stdout:461: done (3 left) 2026-02-25T08:43:44.066 INFO:tasks.rados.rados.0.trial039.stdout:462: done (2 left) 2026-02-25T08:43:44.066 INFO:tasks.rados.rados.0.trial039.stdout:464: done (1 left) 2026-02-25T08:43:44.066 INFO:tasks.rados.rados.0.trial039.stdout:465: done (0 left) 2026-02-25T08:43:44.067 INFO:tasks.rados.rados.0.trial039.stdout:466: read oid 45 snap -1 2026-02-25T08:43:44.067 INFO:tasks.rados.rados.0.trial039.stdout:466: expect (ObjNum 180 snap 40 seq_num 180) 2026-02-25T08:43:44.067 INFO:tasks.rados.rados.0.trial039.stdout:467: snap_remove snap 52 2026-02-25T08:43:45.067 INFO:tasks.rados.rados.0.trial039.stdout:467: done (1 left) 2026-02-25T08:43:45.067 INFO:tasks.rados.rados.0.trial039.stdout:468: delete oid 38 current snap is 62 2026-02-25T08:43:45.067 INFO:tasks.rados.rados.0.trial039.stdout:466: expect (ObjNum 180 snap 40 seq_num 180) 2026-02-25T08:43:45.129 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 515427 2026-02-25T08:43:45.238 INFO:tasks.rados.rados.0.trial039.stdout:466: done (1 left) 2026-02-25T08:43:45.238 INFO:tasks.rados.rados.0.trial039.stdout:468: done (0 left) 2026-02-25T08:43:45.238 INFO:tasks.rados.rados.0.trial039.stdout:469: write (excl) oid 20 current snap is 62 2026-02-25T08:43:45.238 INFO:tasks.rados.rados.0.trial039.stdout:469: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2026-02-25T08:43:45.243 INFO:tasks.rados.rados.0.trial039.stdout:469: writing trial03942481-20 from 648803 to 1371414 tid 1 2026-02-25T08:43:45.248 INFO:tasks.rados.rados.0.trial039.stdout:469: writing trial03942481-20 from 1967200 to 2681947 tid 2 2026-02-25T08:43:45.253 INFO:tasks.rados.rados.0.trial039.stdout:469: writing trial03942481-20 from 3421983 to 3848803 tid 3 2026-02-25T08:43:45.253 INFO:tasks.rados.rados.0.trial039.stdout:470: snap_remove snap 61 2026-02-25T08:43:45.745 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:45.816 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:45.819 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:45.819 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:45.989 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:46.067 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:46.065+0000 7f23c905a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:46.070 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:46.069+0000 7f23c905a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:46.096 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:46.112 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:46.158 INFO:tasks.rados.rados.0.trial039.stdout:470: done (1 left) 2026-02-25T08:43:46.158 INFO:tasks.rados.rados.0.trial039.stdout:471: delete oid 36 current snap is 62 2026-02-25T08:43:46.158 INFO:tasks.rados.rados.0.trial039.stdout:469: finishing write tid 1 to trial03942481-20 2026-02-25T08:43:46.158 INFO:tasks.rados.rados.0.trial039.stdout:469: finishing write tid 2 to trial03942481-20 2026-02-25T08:43:46.158 INFO:tasks.rados.rados.0.trial039.stdout:469: finishing write tid 3 to trial03942481-20 2026-02-25T08:43:46.158 INFO:tasks.rados.rados.0.trial039.stdout:469: finishing write tid 4 to trial03942481-20 2026-02-25T08:43:46.158 INFO:tasks.rados.rados.0.trial039.stdout:469: finishing write tid 5 to trial03942481-20 2026-02-25T08:43:46.158 INFO:tasks.rados.rados.0.trial039.stdout:469: oid 20 updating version 0 to 107 2026-02-25T08:43:46.158 INFO:tasks.rados.rados.0.trial039.stdout:469: oid 20 version 107 is already newer than 107 2026-02-25T08:43:46.158 INFO:tasks.rados.rados.0.trial039.stdout:469: oid 20 version 107 is already newer than 107 2026-02-25T08:43:46.159 INFO:tasks.rados.rados.0.trial039.stdout:469: oid 20 version 107 is already newer than 104 2026-02-25T08:43:46.159 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 20 v 107 (ObjNum 229 snap 62 seq_num 229) dirty exists 2026-02-25T08:43:46.159 INFO:tasks.rados.rados.0.trial039.stdout:469: left oid 20 (ObjNum 229 snap 62 seq_num 229) 2026-02-25T08:43:46.200 INFO:tasks.rados.rados.0.trial039.stdout:469: done (1 left) 2026-02-25T08:43:46.200 INFO:tasks.rados.rados.0.trial039.stdout:471: done (0 left) 2026-02-25T08:43:46.200 INFO:tasks.rados.rados.0.trial039.stdout:472: delete oid 1 current snap is 62 2026-02-25T08:43:46.202 INFO:tasks.rados.rados.0.trial039.stdout:472: done (0 left) 2026-02-25T08:43:46.202 INFO:tasks.rados.rados.0.trial039.stdout:473: read oid 17 snap -1 2026-02-25T08:43:46.202 INFO:tasks.rados.rados.0.trial039.stdout:473: expect (ObjNum 217 snap 56 seq_num 217) 2026-02-25T08:43:46.202 INFO:tasks.rados.rados.0.trial039.stdout:474: read oid 43 snap -1 2026-02-25T08:43:46.202 INFO:tasks.rados.rados.0.trial039.stdout:474: expect (ObjNum 176 snap 40 seq_num 176) 2026-02-25T08:43:46.202 INFO:tasks.rados.rados.0.trial039.stdout:475: read oid 37 snap 59 2026-02-25T08:43:46.203 INFO:tasks.rados.rados.0.trial039.stdout:475: expect deleted 2026-02-25T08:43:46.203 INFO:tasks.rados.rados.0.trial039.stdout:476: read oid 38 snap -1 2026-02-25T08:43:46.203 INFO:tasks.rados.rados.0.trial039.stdout:476: expect deleted 2026-02-25T08:43:46.203 INFO:tasks.rados.rados.0.trial039.stdout:477: snap_create 2026-02-25T08:43:46.218 INFO:tasks.rados.rados.0.trial039.stdout:474: expect (ObjNum 176 snap 40 seq_num 176) 2026-02-25T08:43:46.218 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:46.320 INFO:tasks.rados.rados.0.trial039.stdout:473: expect (ObjNum 217 snap 56 seq_num 217) 2026-02-25T08:43:46.320 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:46.372 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:47.160 INFO:tasks.rados.rados.0.trial039.stdout:473: done (4 left) 2026-02-25T08:43:47.160 INFO:tasks.rados.rados.0.trial039.stdout:474: done (3 left) 2026-02-25T08:43:47.160 INFO:tasks.rados.rados.0.trial039.stdout:475: done (2 left) 2026-02-25T08:43:47.160 INFO:tasks.rados.rados.0.trial039.stdout:476: done (1 left) 2026-02-25T08:43:47.160 INFO:tasks.rados.rados.0.trial039.stdout:477: done (0 left) 2026-02-25T08:43:47.160 INFO:tasks.rados.rados.0.trial039.stdout:478: read oid 9 snap -1 2026-02-25T08:43:47.160 INFO:tasks.rados.rados.0.trial039.stdout:478: expect deleted 2026-02-25T08:43:47.161 INFO:tasks.rados.rados.0.trial039.stdout:479: write (excl) oid 19 current snap is 63 2026-02-25T08:43:47.161 INFO:tasks.rados.rados.0.trial039.stdout:479: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2026-02-25T08:43:47.164 INFO:tasks.rados.rados.0.trial039.stdout:479: writing trial03942481-19 from 699224 to 1247582 tid 1 2026-02-25T08:43:47.169 INFO:tasks.rados.rados.0.trial039.stdout:479: writing trial03942481-19 from 2026293 to 2488367 tid 2 2026-02-25T08:43:47.174 INFO:tasks.rados.rados.0.trial039.stdout:479: writing trial03942481-19 from 3271546 to 3704300 tid 3 2026-02-25T08:43:47.174 INFO:tasks.rados.rados.0.trial039.stdout:480: write (excl) oid 45 current snap is 63 2026-02-25T08:43:47.174 INFO:tasks.rados.rados.0.trial039.stdout:480: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2026-02-25T08:43:47.179 INFO:tasks.rados.rados.0.trial039.stdout:480: writing trial03942481-45 from 716031 to 1339638 tid 1 2026-02-25T08:43:47.184 INFO:tasks.rados.rados.0.trial039.stdout:480: writing trial03942481-45 from 2084775 to 2690507 tid 2 2026-02-25T08:43:47.189 INFO:tasks.rados.rados.0.trial039.stdout:480: writing trial03942481-45 from 3298969 to 3916031 tid 3 2026-02-25T08:43:47.189 INFO:tasks.rados.rados.0.trial039.stdout:481: read oid 16 snap -1 2026-02-25T08:43:47.189 INFO:tasks.rados.rados.0.trial039.stdout:481: expect (ObjNum 153 snap 29 seq_num 153) 2026-02-25T08:43:47.189 INFO:tasks.rados.rados.0.trial039.stdout:482: snap_create 2026-02-25T08:43:47.193 INFO:tasks.rados.rados.0.trial039.stdout:479: finishing write tid 1 to trial03942481-19 2026-02-25T08:43:47.213 INFO:tasks.rados.rados.0.trial039.stdout:479: finishing write tid 2 to trial03942481-19 2026-02-25T08:43:47.214 INFO:tasks.rados.rados.0.trial039.stdout:480: finishing write tid 1 to trial03942481-45 2026-02-25T08:43:47.227 INFO:tasks.rados.rados.0.trial039.stdout:479: finishing write tid 3 to trial03942481-19 2026-02-25T08:43:47.228 INFO:tasks.rados.rados.0.trial039.stdout:481: expect (ObjNum 153 snap 29 seq_num 153) 2026-02-25T08:43:47.251 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:47.288 INFO:tasks.rados.rados.0.trial039.stdout:480: finishing write tid 2 to trial03942481-45 2026-02-25T08:43:47.288 INFO:tasks.rados.rados.0.trial039.stdout:479: finishing write tid 4 to trial03942481-19 2026-02-25T08:43:47.288 INFO:tasks.rados.rados.0.trial039.stdout:479: finishing write tid 5 to trial03942481-19 2026-02-25T08:43:47.288 INFO:tasks.rados.rados.0.trial039.stdout:479: oid 19 updating version 0 to 155 2026-02-25T08:43:47.288 INFO:tasks.rados.rados.0.trial039.stdout:479: oid 19 version 155 is already newer than 152 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:479: oid 19 version 155 is already newer than 155 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:479: oid 19 version 155 is already newer than 155 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 19 v 155 (ObjNum 232 snap 63 seq_num 232) dirty exists 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:479: left oid 19 (ObjNum 232 snap 63 seq_num 232) 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:480: finishing write tid 3 to trial03942481-45 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:480: finishing write tid 4 to trial03942481-45 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:480: finishing write tid 5 to trial03942481-45 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:480: oid 45 updating version 0 to 112 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:480: oid 45 version 112 is already newer than 109 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:480: oid 45 version 112 is already newer than 112 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:480: oid 45 version 112 is already newer than 112 2026-02-25T08:43:47.289 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 45 v 112 (ObjNum 233 snap 63 seq_num 233) dirty exists 2026-02-25T08:43:47.290 INFO:tasks.rados.rados.0.trial039.stdout:480: left oid 45 (ObjNum 233 snap 63 seq_num 233) 2026-02-25T08:43:47.426 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~132s 2026-02-25T08:43:48.113 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:48.162 INFO:tasks.rados.rados.0.trial039.stdout:478: done (4 left) 2026-02-25T08:43:48.163 INFO:tasks.rados.rados.0.trial039.stdout:479: done (3 left) 2026-02-25T08:43:48.163 INFO:tasks.rados.rados.0.trial039.stdout:480: done (2 left) 2026-02-25T08:43:48.163 INFO:tasks.rados.rados.0.trial039.stdout:481: done (1 left) 2026-02-25T08:43:48.163 INFO:tasks.rados.rados.0.trial039.stdout:482: done (0 left) 2026-02-25T08:43:48.163 INFO:tasks.rados.rados.0.trial039.stdout:483: read oid 46 snap -1 2026-02-25T08:43:48.163 INFO:tasks.rados.rados.0.trial039.stdout:483: expect (ObjNum 201 snap 53 seq_num 201) 2026-02-25T08:43:48.163 INFO:tasks.rados.rados.0.trial039.stdout:484: delete oid 32 current snap is 64 2026-02-25T08:43:48.191 INFO:tasks.rados.rados.0.trial039.stdout:483: expect (ObjNum 201 snap 53 seq_num 201) 2026-02-25T08:43:48.196 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:48.193+0000 7f36700c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:48.198 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:48.196+0000 7f36700c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:48.226 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:48.240 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:48.261 INFO:tasks.rados.rados.0.trial039.stdout:483: done (1 left) 2026-02-25T08:43:48.261 INFO:tasks.rados.rados.0.trial039.stdout:484: done (0 left) 2026-02-25T08:43:48.261 INFO:tasks.rados.rados.0.trial039.stdout:485: write (excl) oid 40 current snap is 64 2026-02-25T08:43:48.261 INFO:tasks.rados.rados.0.trial039.stdout:485: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2026-02-25T08:43:48.267 INFO:tasks.rados.rados.0.trial039.stdout:485: writing trial03942481-40 from 749645 to 1523750 tid 1 2026-02-25T08:43:48.271 INFO:tasks.rados.rados.0.trial039.stdout:485: writing trial03942481-40 from 2085386 to 2694787 tid 2 2026-02-25T08:43:48.276 INFO:tasks.rados.rados.0.trial039.stdout:485: writing trial03942481-40 from 3237462 to 3949645 tid 3 2026-02-25T08:43:48.276 INFO:tasks.rados.rados.0.trial039.stdout:486: delete oid 46 current snap is 64 2026-02-25T08:43:48.292 INFO:tasks.rados.rados.0.trial039.stdout:485: finishing write tid 1 to trial03942481-40 2026-02-25T08:43:48.305 INFO:tasks.rados.rados.0.trial039.stdout:486: done (1 left) 2026-02-25T08:43:48.305 INFO:tasks.rados.rados.0.trial039.stdout:487: snap_remove snap 60 2026-02-25T08:43:49.168 INFO:tasks.rados.rados.0.trial039.stdout:487: done (1 left) 2026-02-25T08:43:49.168 INFO:tasks.rados.rados.0.trial039.stdout:488: write oid 29 current snap is 64 2026-02-25T08:43:49.169 INFO:tasks.rados.rados.0.trial039.stdout:485: finishing write tid 2 to trial03942481-40 2026-02-25T08:43:49.169 INFO:tasks.rados.rados.0.trial039.stdout:488: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2026-02-25T08:43:49.174 INFO:tasks.rados.rados.0.trial039.stdout:488: writing trial03942481-29 from 783259 to 1424215 tid 1 2026-02-25T08:43:49.178 INFO:tasks.rados.rados.0.trial039.stdout:488: writing trial03942481-29 from 1918703 to 2531773 tid 2 2026-02-25T08:43:49.181 INFO:tasks.rados.rados.0.trial039.stdout:488: writing trial03942481-29 from 3008661 to 3414128 tid 3 2026-02-25T08:43:49.184 INFO:tasks.rados.rados.0.trial039.stdout:488: writing trial03942481-29 from 3821939 to 3983259 tid 4 2026-02-25T08:43:49.184 INFO:tasks.rados.rados.0.trial039.stdout:489: snap_remove snap 63 2026-02-25T08:43:50.172 INFO:tasks.rados.rados.0.trial039.stdout:489: done (2 left) 2026-02-25T08:43:50.172 INFO:tasks.rados.rados.0.trial039.stdout:490: snap_create 2026-02-25T08:43:50.172 INFO:tasks.rados.rados.0.trial039.stdout:485: finishing write tid 3 to trial03942481-40 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:485: finishing write tid 4 to trial03942481-40 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:485: finishing write tid 5 to trial03942481-40 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:485: oid 40 updating version 0 to 116 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:485: oid 40 version 116 is already newer than 113 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:485: oid 40 version 116 is already newer than 116 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:485: oid 40 version 116 is already newer than 116 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 40 v 116 (ObjNum 235 snap 64 seq_num 235) dirty exists 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:485: left oid 40 (ObjNum 235 snap 64 seq_num 235) 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:488: finishing write tid 1 to trial03942481-29 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:488: finishing write tid 2 to trial03942481-29 2026-02-25T08:43:50.173 INFO:tasks.rados.rados.0.trial039.stdout:488: finishing write tid 3 to trial03942481-29 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial039.stdout:488: finishing write tid 4 to trial03942481-29 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial039.stdout:488: finishing write tid 5 to trial03942481-29 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial039.stdout:488: finishing write tid 6 to trial03942481-29 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial039.stdout:488: oid 29 updating version 0 to 39 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial039.stdout:488: oid 29 version 39 is already newer than 39 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial039.stdout:488: oid 29 version 39 is already newer than 39 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial039.stdout:488: oid 29 version 39 is already newer than 39 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial039.stdout:488: oid 29 version 39 is already newer than 39 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 29 v 39 (ObjNum 237 snap 64 seq_num 237) dirty exists 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial039.stdout:488: left oid 29 (ObjNum 237 snap 64 seq_num 237) 2026-02-25T08:43:50.241 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:50.317 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:50.313+0000 7f51b0d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:50.320 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:50.317+0000 7f51b0d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:50.345 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:50.360 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:50.820 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:50.892 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:50.896 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:51.174 INFO:tasks.rados.rados.0.trial039.stdout:485: done (2 left) 2026-02-25T08:43:51.174 INFO:tasks.rados.rados.0.trial039.stdout:488: done (1 left) 2026-02-25T08:43:51.174 INFO:tasks.rados.rados.0.trial039.stdout:490: done (0 left) 2026-02-25T08:43:51.175 INFO:tasks.rados.rados.0.trial039.stdout:491: read oid 19 snap -1 2026-02-25T08:43:51.175 INFO:tasks.rados.rados.0.trial039.stdout:491: expect (ObjNum 232 snap 63 seq_num 232) 2026-02-25T08:43:51.175 INFO:tasks.rados.rados.0.trial039.stdout:492: delete oid 3 current snap is 65 2026-02-25T08:43:51.209 INFO:tasks.rados.rados.0.trial039.stdout:491: expect (ObjNum 232 snap 63 seq_num 232) 2026-02-25T08:43:51.218 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2402697 2026-02-25T08:43:51.293 INFO:tasks.rados.rados.0.trial039.stdout:491: done (1 left) 2026-02-25T08:43:51.293 INFO:tasks.rados.rados.0.trial039.stdout:492: done (0 left) 2026-02-25T08:43:51.293 INFO:tasks.rados.rados.0.trial039.stdout:493: read oid 22 snap 53 2026-02-25T08:43:51.294 INFO:tasks.rados.rados.0.trial039.stdout:493: expect (ObjNum 200 snap 51 seq_num 200) 2026-02-25T08:43:51.294 INFO:tasks.rados.rados.0.trial039.stdout:494: read oid 12 snap -1 2026-02-25T08:43:51.294 INFO:tasks.rados.rados.0.trial039.stdout:494: expect (ObjNum 210 snap 55 seq_num 210) 2026-02-25T08:43:51.294 INFO:tasks.rados.rados.0.trial039.stdout:495: read oid 49 snap -1 2026-02-25T08:43:51.294 INFO:tasks.rados.rados.0.trial039.stdout:495: expect deleted 2026-02-25T08:43:51.294 INFO:tasks.rados.rados.0.trial039.stdout:496: read oid 6 snap -1 2026-02-25T08:43:51.294 INFO:tasks.rados.rados.0.trial039.stdout:496: expect (ObjNum 199 snap 50 seq_num 199) 2026-02-25T08:43:51.294 INFO:tasks.rados.rados.0.trial039.stdout:497: snap_remove snap 59 2026-02-25T08:43:52.180 INFO:tasks.rados.rados.0.trial039.stdout:497: done (4 left) 2026-02-25T08:43:52.181 INFO:tasks.rados.rados.0.trial039.stdout:498: delete oid 37 current snap is 65 2026-02-25T08:43:52.181 INFO:tasks.rados.rados.0.trial039.stdout:494: expect (ObjNum 210 snap 55 seq_num 210) 2026-02-25T08:43:52.254 INFO:tasks.rados.rados.0.trial039.stdout:493: expect (ObjNum 200 snap 51 seq_num 200) 2026-02-25T08:43:52.282 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:52.311 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:52.340 INFO:tasks.rados.rados.0.trial039.stdout:496: expect (ObjNum 199 snap 50 seq_num 199) 2026-02-25T08:43:52.340 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:52.361 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:52.437 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:52.434+0000 7ffa044cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:52.440 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:52.437+0000 7ffa044cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:52.468 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:52.484 INFO:tasks.rados.rados.0.trial039.stdout:493: done (4 left) 2026-02-25T08:43:52.484 INFO:tasks.rados.rados.0.trial039.stdout:494: done (3 left) 2026-02-25T08:43:52.484 INFO:tasks.rados.rados.0.trial039.stdout:495: done (2 left) 2026-02-25T08:43:52.484 INFO:tasks.rados.rados.0.trial039.stdout:496: done (1 left) 2026-02-25T08:43:52.484 INFO:tasks.rados.rados.0.trial039.stdout:498: done (0 left) 2026-02-25T08:43:52.484 INFO:tasks.rados.rados.0.trial039.stdout:499: read oid 22 snap -1 2026-02-25T08:43:52.484 INFO:tasks.rados.rados.0.trial039.stdout:499: expect (ObjNum 214 snap 55 seq_num 214) 2026-02-25T08:43:52.484 INFO:tasks.rados.rados.0.trial039.stdout:500: write oid 29 current snap is 65 2026-02-25T08:43:52.484 INFO:tasks.rados.rados.0.trial039.stdout:500: seq_num 240 ranges {433680=466703,1577796=455884} 2026-02-25T08:43:52.485 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:52.485 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:52.487 INFO:tasks.rados.rados.0.trial039.stdout:500: writing trial03942481-29 from 433680 to 900383 tid 1 2026-02-25T08:43:52.491 INFO:tasks.rados.rados.0.trial039.stdout:500: writing trial03942481-29 from 1577796 to 2033680 tid 2 2026-02-25T08:43:52.491 INFO:tasks.rados.rados.0.trial039.stdout:501: snap_remove snap 49 2026-02-25T08:43:52.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~138s 2026-02-25T08:43:53.067 INFO:tasks.rados.rados.0.trial039.stdout:501: done (2 left) 2026-02-25T08:43:53.067 INFO:tasks.rados.rados.0.trial039.stdout:502: write (excl) oid 11 current snap is 65 2026-02-25T08:43:53.068 INFO:tasks.rados.rados.0.trial039.stdout:502: seq_num 241 ranges {450487=541952,1636278=414209} 2026-02-25T08:43:53.071 INFO:tasks.rados.rados.0.trial039.stdout:502: writing trial03942481-11 from 450487 to 992439 tid 1 2026-02-25T08:43:53.076 INFO:tasks.rados.rados.0.trial039.stdout:502: writing trial03942481-11 from 1636278 to 2050487 tid 2 2026-02-25T08:43:53.076 INFO:tasks.rados.rados.0.trial039.stdout:503: read oid 39 snap -1 2026-02-25T08:43:53.076 INFO:tasks.rados.rados.0.trial039.stdout:503: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:43:53.077 INFO:tasks.rados.rados.0.trial039.stdout:504: read oid 45 snap -1 2026-02-25T08:43:53.077 INFO:tasks.rados.rados.0.trial039.stdout:504: expect (ObjNum 233 snap 63 seq_num 233) 2026-02-25T08:43:53.077 INFO:tasks.rados.rados.0.trial039.stdout:505: snap_remove snap 54 2026-02-25T08:43:54.068 INFO:tasks.rados.rados.0.trial039.stdout:505: done (5 left) 2026-02-25T08:43:54.069 INFO:tasks.rados.rados.0.trial039.stdout:506: snap_create 2026-02-25T08:43:54.069 INFO:tasks.rados.rados.0.trial039.stdout:499: expect (ObjNum 214 snap 55 seq_num 214) 2026-02-25T08:43:54.069 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:54.141 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 727603 2026-02-25T08:43:54.175 INFO:tasks.rados.rados.0.trial039.stdout:500: finishing write tid 1 to trial03942481-29 2026-02-25T08:43:54.175 INFO:tasks.rados.rados.0.trial039.stdout:500: finishing write tid 2 to trial03942481-29 2026-02-25T08:43:54.175 INFO:tasks.rados.rados.0.trial039.stdout:500: finishing write tid 3 to trial03942481-29 2026-02-25T08:43:54.175 INFO:tasks.rados.rados.0.trial039.stdout:500: finishing write tid 4 to trial03942481-29 2026-02-25T08:43:54.175 INFO:tasks.rados.rados.0.trial039.stdout:500: oid 29 updating version 0 to 43 2026-02-25T08:43:54.175 INFO:tasks.rados.rados.0.trial039.stdout:500: oid 29 version 43 is already newer than 43 2026-02-25T08:43:54.175 INFO:tasks.rados.rados.0.trial039.stdout:500: oid 29 version 43 is already newer than 43 2026-02-25T08:43:54.175 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 29 v 43 (ObjNum 240 snap 65 seq_num 240) dirty exists 2026-02-25T08:43:54.176 INFO:tasks.rados.rados.0.trial039.stdout:500: left oid 29 (ObjNum 240 snap 65 seq_num 240) 2026-02-25T08:43:54.176 INFO:tasks.rados.rados.0.trial039.stdout:503: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:43:54.176 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:54.199 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1847758 2026-02-25T08:43:54.232 INFO:tasks.rados.rados.0.trial039.stdout:504: expect (ObjNum 233 snap 63 seq_num 233) 2026-02-25T08:43:54.310 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2007182 2026-02-25T08:43:54.401 INFO:tasks.rados.rados.0.trial039.stdout:502: finishing write tid 1 to trial03942481-11 2026-02-25T08:43:54.402 INFO:tasks.rados.rados.0.trial039.stdout:502: finishing write tid 2 to trial03942481-11 2026-02-25T08:43:54.402 INFO:tasks.rados.rados.0.trial039.stdout:502: finishing write tid 3 to trial03942481-11 2026-02-25T08:43:54.402 INFO:tasks.rados.rados.0.trial039.stdout:502: finishing write tid 4 to trial03942481-11 2026-02-25T08:43:54.402 INFO:tasks.rados.rados.0.trial039.stdout:502: oid 11 updating version 0 to 18 2026-02-25T08:43:54.402 INFO:tasks.rados.rados.0.trial039.stdout:502: oid 11 updating version 18 to 20 2026-02-25T08:43:54.402 INFO:tasks.rados.rados.0.trial039.stdout:502: oid 11 version 20 is already newer than 20 2026-02-25T08:43:54.402 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 11 v 20 (ObjNum 241 snap 65 seq_num 241) dirty exists 2026-02-25T08:43:54.402 INFO:tasks.rados.rados.0.trial039.stdout:502: left oid 11 (ObjNum 241 snap 65 seq_num 241) 2026-02-25T08:43:54.486 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:54.562 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:54.559+0000 7fc9657dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:54.565 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:54.562+0000 7fc9657dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:54.591 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:54.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:55.069 INFO:tasks.rados.rados.0.trial039.stdout:499: done (5 left) 2026-02-25T08:43:55.070 INFO:tasks.rados.rados.0.trial039.stdout:500: done (4 left) 2026-02-25T08:43:55.070 INFO:tasks.rados.rados.0.trial039.stdout:502: done (3 left) 2026-02-25T08:43:55.070 INFO:tasks.rados.rados.0.trial039.stdout:503: done (2 left) 2026-02-25T08:43:55.070 INFO:tasks.rados.rados.0.trial039.stdout:504: done (1 left) 2026-02-25T08:43:55.070 INFO:tasks.rados.rados.0.trial039.stdout:506: done (0 left) 2026-02-25T08:43:55.070 INFO:tasks.rados.rados.0.trial039.stdout:507: delete oid 2 current snap is 66 2026-02-25T08:43:55.111 INFO:tasks.rados.rados.0.trial039.stdout:507: done (0 left) 2026-02-25T08:43:55.111 INFO:tasks.rados.rados.0.trial039.stdout:508: read oid 14 snap -1 2026-02-25T08:43:55.111 INFO:tasks.rados.rados.0.trial039.stdout:508: expect (ObjNum 208 snap 55 seq_num 208) 2026-02-25T08:43:55.111 INFO:tasks.rados.rados.0.trial039.stdout:509: write oid 31 current snap is 66 2026-02-25T08:43:55.111 INFO:tasks.rados.rados.0.trial039.stdout:509: seq_num 243 ranges {484101=692450,1636889=447212} 2026-02-25T08:43:55.115 INFO:tasks.rados.rados.0.trial039.stdout:509: writing trial03942481-31 from 484101 to 1176551 tid 1 2026-02-25T08:43:55.119 INFO:tasks.rados.rados.0.trial039.stdout:509: writing trial03942481-31 from 1636889 to 2084101 tid 2 2026-02-25T08:43:55.120 INFO:tasks.rados.rados.0.trial039.stdout:510: snap_create 2026-02-25T08:43:55.144 INFO:tasks.rados.rados.0.trial039.stdout:508: expect (ObjNum 208 snap 55 seq_num 208) 2026-02-25T08:43:55.200 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:55.254 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:55.308 INFO:tasks.rados.rados.0.trial039.stdout:509: finishing write tid 1 to trial03942481-31 2026-02-25T08:43:55.308 INFO:tasks.rados.rados.0.trial039.stdout:509: finishing write tid 2 to trial03942481-31 2026-02-25T08:43:55.308 INFO:tasks.rados.rados.0.trial039.stdout:509: finishing write tid 3 to trial03942481-31 2026-02-25T08:43:55.308 INFO:tasks.rados.rados.0.trial039.stdout:509: finishing write tid 4 to trial03942481-31 2026-02-25T08:43:55.309 INFO:tasks.rados.rados.0.trial039.stdout:509: oid 31 updating version 0 to 171 2026-02-25T08:43:55.309 INFO:tasks.rados.rados.0.trial039.stdout:509: oid 31 version 171 is already newer than 171 2026-02-25T08:43:55.309 INFO:tasks.rados.rados.0.trial039.stdout:509: oid 31 version 171 is already newer than 171 2026-02-25T08:43:55.309 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 171 (ObjNum 243 snap 66 seq_num 243) dirty exists 2026-02-25T08:43:55.309 INFO:tasks.rados.rados.0.trial039.stdout:509: left oid 31 (ObjNum 243 snap 66 seq_num 243) 2026-02-25T08:43:55.897 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:43:55.971 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:55.975 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:43:56.076 INFO:tasks.rados.rados.0.trial039.stdout:508: done (2 left) 2026-02-25T08:43:56.076 INFO:tasks.rados.rados.0.trial039.stdout:509: done (1 left) 2026-02-25T08:43:56.076 INFO:tasks.rados.rados.0.trial039.stdout:510: done (0 left) 2026-02-25T08:43:56.076 INFO:tasks.rados.rados.0.trial039.stdout:511: read oid 15 snap 51 2026-02-25T08:43:56.076 INFO:tasks.rados.rados.0.trial039.stdout:511: expect deleted 2026-02-25T08:43:56.076 INFO:tasks.rados.rados.0.trial039.stdout:512: read oid 43 snap 57 2026-02-25T08:43:56.076 INFO:tasks.rados.rados.0.trial039.stdout:512: expect (ObjNum 176 snap 40 seq_num 176) 2026-02-25T08:43:56.076 INFO:tasks.rados.rados.0.trial039.stdout:513: write oid 16 current snap is 67 2026-02-25T08:43:56.077 INFO:tasks.rados.rados.0.trial039.stdout:513: seq_num 244 ranges {500908=484052,1411724=689184} 2026-02-25T08:43:56.079 INFO:tasks.rados.rados.0.trial039.stdout:513: writing trial03942481-16 from 500908 to 984960 tid 1 2026-02-25T08:43:56.083 INFO:tasks.rados.rados.0.trial039.stdout:513: writing trial03942481-16 from 1411724 to 2100908 tid 2 2026-02-25T08:43:56.083 INFO:tasks.rados.rados.0.trial039.stdout:514: delete oid 2 current snap is 67 2026-02-25T08:43:56.093 INFO:tasks.rados.rados.0.trial039.stdout:514: done (3 left) 2026-02-25T08:43:56.093 INFO:tasks.rados.rados.0.trial039.stdout:515: snap_remove snap 62 2026-02-25T08:43:56.609 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:56.688 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:56.684+0000 7fc489d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:56.690 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:56.688+0000 7fc489d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:56.719 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:56.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:57.079 INFO:tasks.rados.rados.0.trial039.stdout:515: done (3 left) 2026-02-25T08:43:57.079 INFO:tasks.rados.rados.0.trial039.stdout:516: read oid 26 snap -1 2026-02-25T08:43:57.079 INFO:tasks.rados.rados.0.trial039.stdout:516: expect (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:43:57.079 INFO:tasks.rados.rados.0.trial039.stdout:517: snap_create 2026-02-25T08:43:57.079 INFO:tasks.rados.rados.0.trial039.stdout:512: expect (ObjNum 176 snap 40 seq_num 176) 2026-02-25T08:43:57.127 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2719856 2026-02-25T08:43:57.164 INFO:tasks.rados.rados.0.trial039.stdout:513: finishing write tid 1 to trial03942481-16 2026-02-25T08:43:57.164 INFO:tasks.rados.rados.0.trial039.stdout:513: finishing write tid 2 to trial03942481-16 2026-02-25T08:43:57.165 INFO:tasks.rados.rados.0.trial039.stdout:513: finishing write tid 3 to trial03942481-16 2026-02-25T08:43:57.165 INFO:tasks.rados.rados.0.trial039.stdout:513: finishing write tid 4 to trial03942481-16 2026-02-25T08:43:57.165 INFO:tasks.rados.rados.0.trial039.stdout:513: oid 16 updating version 0 to 37 2026-02-25T08:43:57.165 INFO:tasks.rados.rados.0.trial039.stdout:513: oid 16 version 37 is already newer than 37 2026-02-25T08:43:57.165 INFO:tasks.rados.rados.0.trial039.stdout:513: oid 16 version 37 is already newer than 37 2026-02-25T08:43:57.165 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 16 v 37 (ObjNum 244 snap 67 seq_num 244) dirty exists 2026-02-25T08:43:57.165 INFO:tasks.rados.rados.0.trial039.stdout:513: left oid 16 (ObjNum 244 snap 67 seq_num 244) 2026-02-25T08:43:57.165 INFO:tasks.rados.rados.0.trial039.stdout:516: expect (ObjNum 134 snap 21 seq_num 134) 2026-02-25T08:43:57.165 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:43:58.078 INFO:tasks.rados.rados.0.trial039.stdout:511: done (4 left) 2026-02-25T08:43:58.078 INFO:tasks.rados.rados.0.trial039.stdout:512: done (3 left) 2026-02-25T08:43:58.078 INFO:tasks.rados.rados.0.trial039.stdout:513: done (2 left) 2026-02-25T08:43:58.078 INFO:tasks.rados.rados.0.trial039.stdout:516: done (1 left) 2026-02-25T08:43:58.078 INFO:tasks.rados.rados.0.trial039.stdout:517: done (0 left) 2026-02-25T08:43:58.079 INFO:tasks.rados.rados.0.trial039.stdout:518: write oid 12 current snap is 68 2026-02-25T08:43:58.079 INFO:tasks.rados.rados.0.trial039.stdout:518: seq_num 246 ranges {534522=634550,1812335=322187} 2026-02-25T08:43:58.082 INFO:tasks.rados.rados.0.trial039.stdout:518: writing trial03942481-12 from 534522 to 1169072 tid 1 2026-02-25T08:43:58.086 INFO:tasks.rados.rados.0.trial039.stdout:518: writing trial03942481-12 from 1812335 to 2134522 tid 2 2026-02-25T08:43:58.086 INFO:tasks.rados.rados.0.trial039.stdout:519: snap_remove snap 51 2026-02-25T08:43:58.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~143s 2026-02-25T08:43:58.737 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:58.812 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:58.809+0000 7f839a10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:58.815 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:43:58.813+0000 7f839a10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:58.841 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:43:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:58.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:59.083 INFO:tasks.rados.rados.0.trial039.stdout:519: done (1 left) 2026-02-25T08:43:59.083 INFO:tasks.rados.rados.0.trial039.stdout:520: delete oid 31 current snap is 68 2026-02-25T08:43:59.083 INFO:tasks.rados.rados.0.trial039.stdout:518: finishing write tid 1 to trial03942481-12 2026-02-25T08:43:59.083 INFO:tasks.rados.rados.0.trial039.stdout:518: finishing write tid 2 to trial03942481-12 2026-02-25T08:43:59.083 INFO:tasks.rados.rados.0.trial039.stdout:518: finishing write tid 3 to trial03942481-12 2026-02-25T08:43:59.084 INFO:tasks.rados.rados.0.trial039.stdout:518: finishing write tid 4 to trial03942481-12 2026-02-25T08:43:59.084 INFO:tasks.rados.rados.0.trial039.stdout:518: oid 12 updating version 0 to 41 2026-02-25T08:43:59.084 INFO:tasks.rados.rados.0.trial039.stdout:518: oid 12 version 41 is already newer than 41 2026-02-25T08:43:59.084 INFO:tasks.rados.rados.0.trial039.stdout:518: oid 12 version 41 is already newer than 41 2026-02-25T08:43:59.084 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 12 v 41 (ObjNum 246 snap 68 seq_num 246) dirty exists 2026-02-25T08:43:59.084 INFO:tasks.rados.rados.0.trial039.stdout:518: left oid 12 (ObjNum 246 snap 68 seq_num 246) 2026-02-25T08:43:59.121 INFO:tasks.rados.rados.0.trial039.stdout:518: done (1 left) 2026-02-25T08:43:59.122 INFO:tasks.rados.rados.0.trial039.stdout:520: done (0 left) 2026-02-25T08:43:59.122 INFO:tasks.rados.rados.0.trial039.stdout:521: write (excl) oid 50 current snap is 68 2026-02-25T08:43:59.122 INFO:tasks.rados.rados.0.trial039.stdout:521: seq_num 248 ranges {568136=785048,1929299=238837} 2026-02-25T08:43:59.126 INFO:tasks.rados.rados.0.trial039.stdout:521: writing trial03942481-50 from 568136 to 1353184 tid 1 2026-02-25T08:43:59.129 INFO:tasks.rados.rados.0.trial039.stdout:521: writing trial03942481-50 from 1929299 to 2168136 tid 2 2026-02-25T08:43:59.129 INFO:tasks.rados.rados.0.trial039.stdout:522: snap_create 2026-02-25T08:43:59.159 INFO:tasks.rados.rados.0.trial039.stdout:521: finishing write tid 1 to trial03942481-50 2026-02-25T08:43:59.172 INFO:tasks.rados.rados.0.trial039.stdout:521: finishing write tid 2 to trial03942481-50 2026-02-25T08:43:59.180 INFO:tasks.rados.rados.0.trial039.stdout:521: finishing write tid 3 to trial03942481-50 2026-02-25T08:43:59.181 INFO:tasks.rados.rados.0.trial039.stdout:521: finishing write tid 4 to trial03942481-50 2026-02-25T08:43:59.182 INFO:tasks.rados.rados.0.trial039.stdout:521: oid 50 updating version 0 to 118 2026-02-25T08:43:59.182 INFO:tasks.rados.rados.0.trial039.stdout:521: oid 50 updating version 118 to 120 2026-02-25T08:43:59.182 INFO:tasks.rados.rados.0.trial039.stdout:521: oid 50 version 120 is already newer than 120 2026-02-25T08:43:59.182 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 50 v 120 (ObjNum 248 snap 68 seq_num 248) dirty exists 2026-02-25T08:43:59.182 INFO:tasks.rados.rados.0.trial039.stdout:521: left oid 50 (ObjNum 248 snap 68 seq_num 248) 2026-02-25T08:44:00.088 INFO:tasks.rados.rados.0.trial039.stdout:521: done (1 left) 2026-02-25T08:44:00.089 INFO:tasks.rados.rados.0.trial039.stdout:522: done (0 left) 2026-02-25T08:44:00.089 INFO:tasks.rados.rados.0.trial039.stdout:523: read oid 17 snap -1 2026-02-25T08:44:00.089 INFO:tasks.rados.rados.0.trial039.stdout:523: expect (ObjNum 217 snap 56 seq_num 217) 2026-02-25T08:44:00.089 INFO:tasks.rados.rados.0.trial039.stdout:524: snap_create 2026-02-25T08:44:00.117 INFO:tasks.rados.rados.0.trial039.stdout:523: expect (ObjNum 217 snap 56 seq_num 217) 2026-02-25T08:44:00.118 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:00.186 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1746180 2026-02-25T08:44:00.859 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:00.937 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:00.934+0000 7f7310ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:00.939 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:00.937+0000 7f7310ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:00.964 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:00.976 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:00.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:01.046 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:01.049 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:01.049 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:01.092 INFO:tasks.rados.rados.0.trial039.stdout:523: done (1 left) 2026-02-25T08:44:01.092 INFO:tasks.rados.rados.0.trial039.stdout:524: done (0 left) 2026-02-25T08:44:01.092 INFO:tasks.rados.rados.0.trial039.stdout:525: write oid 36 current snap is 70 2026-02-25T08:44:01.092 INFO:tasks.rados.rados.0.trial039.stdout:525: seq_num 249 ranges {584943=460297,1587781=597162} 2026-02-25T08:44:01.095 INFO:tasks.rados.rados.0.trial039.stdout:525: writing trial03942481-36 from 584943 to 1045240 tid 1 2026-02-25T08:44:01.100 INFO:tasks.rados.rados.0.trial039.stdout:525: writing trial03942481-36 from 1587781 to 2184943 tid 2 2026-02-25T08:44:01.100 INFO:tasks.rados.rados.0.trial039.stdout:526: read oid 31 snap -1 2026-02-25T08:44:01.100 INFO:tasks.rados.rados.0.trial039.stdout:526: expect deleted 2026-02-25T08:44:01.100 INFO:tasks.rados.rados.0.trial039.stdout:527: write (excl) oid 1 current snap is 70 2026-02-25T08:44:01.100 INFO:tasks.rados.rados.0.trial039.stdout:527: seq_num 250 ranges {601750=535546,1929910=271840} 2026-02-25T08:44:01.104 INFO:tasks.rados.rados.0.trial039.stdout:527: writing trial03942481-1 from 601750 to 1137296 tid 1 2026-02-25T08:44:01.108 INFO:tasks.rados.rados.0.trial039.stdout:527: writing trial03942481-1 from 1929910 to 2201750 tid 2 2026-02-25T08:44:01.108 INFO:tasks.rados.rados.0.trial039.stdout:528: delete oid 14 current snap is 70 2026-02-25T08:44:01.138 INFO:tasks.rados.rados.0.trial039.stdout:527: finishing write tid 1 to trial03942481-1 2026-02-25T08:44:01.158 INFO:tasks.rados.rados.0.trial039.stdout:527: finishing write tid 2 to trial03942481-1 2026-02-25T08:44:01.169 INFO:tasks.rados.rados.0.trial039.stdout:527: finishing write tid 3 to trial03942481-1 2026-02-25T08:44:01.173 INFO:tasks.rados.rados.0.trial039.stdout:527: finishing write tid 4 to trial03942481-1 2026-02-25T08:44:01.173 INFO:tasks.rados.rados.0.trial039.stdout:527: oid 1 updating version 0 to 46 2026-02-25T08:44:01.173 INFO:tasks.rados.rados.0.trial039.stdout:527: oid 1 version 46 is already newer than 44 2026-02-25T08:44:01.173 INFO:tasks.rados.rados.0.trial039.stdout:527: oid 1 version 46 is already newer than 46 2026-02-25T08:44:01.173 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 1 v 46 (ObjNum 250 snap 70 seq_num 250) dirty exists 2026-02-25T08:44:01.173 INFO:tasks.rados.rados.0.trial039.stdout:527: left oid 1 (ObjNum 250 snap 70 seq_num 250) 2026-02-25T08:44:01.184 INFO:tasks.rados.rados.0.trial039.stdout:525: finishing write tid 1 to trial03942481-36 2026-02-25T08:44:01.195 INFO:tasks.rados.rados.0.trial039.stdout:526: done (3 left) 2026-02-25T08:44:01.195 INFO:tasks.rados.rados.0.trial039.stdout:527: done (2 left) 2026-02-25T08:44:01.195 INFO:tasks.rados.rados.0.trial039.stdout:528: done (1 left) 2026-02-25T08:44:01.195 INFO:tasks.rados.rados.0.trial039.stdout:529: write oid 44 current snap is 70 2026-02-25T08:44:01.196 INFO:tasks.rados.rados.0.trial039.stdout:529: seq_num 252 ranges {635364=402397,1763227=472137} 2026-02-25T08:44:01.199 INFO:tasks.rados.rados.0.trial039.stdout:529: writing trial03942481-44 from 635364 to 1037761 tid 1 2026-02-25T08:44:01.204 INFO:tasks.rados.rados.0.trial039.stdout:529: writing trial03942481-44 from 1763227 to 2235364 tid 2 2026-02-25T08:44:01.204 INFO:tasks.rados.rados.0.trial039.stdout:530: read oid 27 snap 67 2026-02-25T08:44:01.204 INFO:tasks.rados.rados.0.trial039.stdout:530: expect (ObjNum 175 snap 40 seq_num 175) 2026-02-25T08:44:01.204 INFO:tasks.rados.rados.0.trial039.stdout:531: delete oid 13 current snap is 70 2026-02-25T08:44:01.204 INFO:tasks.rados.rados.0.trial039.stdout:525: finishing write tid 2 to trial03942481-36 2026-02-25T08:44:01.204 INFO:tasks.rados.rados.0.trial039.stdout:525: finishing write tid 3 to trial03942481-36 2026-02-25T08:44:01.207 INFO:tasks.rados.rados.0.trial039.stdout:531: done (3 left) 2026-02-25T08:44:01.207 INFO:tasks.rados.rados.0.trial039.stdout:532: snap_create 2026-02-25T08:44:01.212 INFO:tasks.rados.rados.0.trial039.stdout:525: finishing write tid 4 to trial03942481-36 2026-02-25T08:44:01.212 INFO:tasks.rados.rados.0.trial039.stdout:525: oid 36 updating version 0 to 68 2026-02-25T08:44:01.212 INFO:tasks.rados.rados.0.trial039.stdout:525: oid 36 version 68 is already newer than 68 2026-02-25T08:44:01.212 INFO:tasks.rados.rados.0.trial039.stdout:525: oid 36 version 68 is already newer than 68 2026-02-25T08:44:01.212 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 36 v 68 (ObjNum 249 snap 70 seq_num 249) dirty exists 2026-02-25T08:44:01.212 INFO:tasks.rados.rados.0.trial039.stdout:525: left oid 36 (ObjNum 249 snap 70 seq_num 249) 2026-02-25T08:44:01.227 INFO:tasks.rados.rados.0.trial039.stdout:530: expect (ObjNum 175 snap 40 seq_num 175) 2026-02-25T08:44:01.242 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2026201 2026-02-25T08:44:01.269 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2749092 2026-02-25T08:44:01.301 INFO:tasks.rados.rados.0.trial039.stdout:529: finishing write tid 1 to trial03942481-44 2026-02-25T08:44:01.301 INFO:tasks.rados.rados.0.trial039.stdout:529: finishing write tid 2 to trial03942481-44 2026-02-25T08:44:01.301 INFO:tasks.rados.rados.0.trial039.stdout:529: finishing write tid 3 to trial03942481-44 2026-02-25T08:44:01.301 INFO:tasks.rados.rados.0.trial039.stdout:529: finishing write tid 4 to trial03942481-44 2026-02-25T08:44:01.301 INFO:tasks.rados.rados.0.trial039.stdout:529: oid 44 updating version 0 to 97 2026-02-25T08:44:01.301 INFO:tasks.rados.rados.0.trial039.stdout:529: oid 44 version 97 is already newer than 97 2026-02-25T08:44:01.301 INFO:tasks.rados.rados.0.trial039.stdout:529: oid 44 version 97 is already newer than 97 2026-02-25T08:44:01.301 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 97 (ObjNum 252 snap 70 seq_num 252) dirty exists 2026-02-25T08:44:01.301 INFO:tasks.rados.rados.0.trial039.stdout:529: left oid 44 (ObjNum 252 snap 70 seq_num 252) 2026-02-25T08:44:02.098 INFO:tasks.rados.rados.0.trial039.stdout:525: done (3 left) 2026-02-25T08:44:02.098 INFO:tasks.rados.rados.0.trial039.stdout:529: done (2 left) 2026-02-25T08:44:02.099 INFO:tasks.rados.rados.0.trial039.stdout:530: done (1 left) 2026-02-25T08:44:02.099 INFO:tasks.rados.rados.0.trial039.stdout:532: done (0 left) 2026-02-25T08:44:02.099 INFO:tasks.rados.rados.0.trial039.stdout:533: snap_remove snap 69 2026-02-25T08:44:02.981 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:03.059 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:03.056+0000 7fc3981c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:03.061 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:03.060+0000 7fc3981c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:03.067 INFO:tasks.rados.rados.0.trial039.stdout:533: done (0 left) 2026-02-25T08:44:03.067 INFO:tasks.rados.rados.0.trial039.stdout:534: snap_remove snap 53 2026-02-25T08:44:03.100 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:03.116 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:03.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~149s 2026-02-25T08:44:04.068 INFO:tasks.rados.rados.0.trial039.stdout:534: done (0 left) 2026-02-25T08:44:04.068 INFO:tasks.rados.rados.0.trial039.stdout:535: write oid 26 current snap is 71 2026-02-25T08:44:04.068 INFO:tasks.rados.rados.0.trial039.stdout:535: seq_num 254 ranges {668978=552895,1763838=505140} 2026-02-25T08:44:04.072 INFO:tasks.rados.rados.0.trial039.stdout:535: writing trial03942481-26 from 668978 to 1221873 tid 1 2026-02-25T08:44:04.077 INFO:tasks.rados.rados.0.trial039.stdout:535: writing trial03942481-26 from 1763838 to 2268978 tid 2 2026-02-25T08:44:04.077 INFO:tasks.rados.rados.0.trial039.stdout:536: read oid 29 snap -1 2026-02-25T08:44:04.077 INFO:tasks.rados.rados.0.trial039.stdout:536: expect (ObjNum 240 snap 65 seq_num 240) 2026-02-25T08:44:04.077 INFO:tasks.rados.rados.0.trial039.stdout:537: write (excl) oid 41 current snap is 71 2026-02-25T08:44:04.077 INFO:tasks.rados.rados.0.trial039.stdout:537: seq_num 255 ranges {685785=628144,1822320=463465} 2026-02-25T08:44:04.082 INFO:tasks.rados.rados.0.trial039.stdout:537: writing trial03942481-41 from 685785 to 1313929 tid 1 2026-02-25T08:44:04.086 INFO:tasks.rados.rados.0.trial039.stdout:537: writing trial03942481-41 from 1822320 to 2285785 tid 2 2026-02-25T08:44:04.086 INFO:tasks.rados.rados.0.trial039.stdout:538: write oid 42 current snap is 71 2026-02-25T08:44:04.087 INFO:tasks.rados.rados.0.trial039.stdout:538: seq_num 256 ranges {702592=703393,1880802=421790} 2026-02-25T08:44:04.091 INFO:tasks.rados.rados.0.trial039.stdout:538: writing trial03942481-42 from 702592 to 1405985 tid 1 2026-02-25T08:44:04.095 INFO:tasks.rados.rados.0.trial039.stdout:538: writing trial03942481-42 from 1880802 to 2302592 tid 2 2026-02-25T08:44:04.095 INFO:tasks.rados.rados.0.trial039.stdout:539: write (excl) oid 46 current snap is 71 2026-02-25T08:44:04.095 INFO:tasks.rados.rados.0.trial039.stdout:539: seq_num 257 ranges {719399=778642,1939284=380115} 2026-02-25T08:44:04.100 INFO:tasks.rados.rados.0.trial039.stdout:539: writing trial03942481-46 from 719399 to 1498041 tid 1 2026-02-25T08:44:04.104 INFO:tasks.rados.rados.0.trial039.stdout:539: writing trial03942481-46 from 1939284 to 2319399 tid 2 2026-02-25T08:44:04.104 INFO:tasks.rados.rados.0.trial039.stdout:540: write (excl) oid 17 current snap is 71 2026-02-25T08:44:04.104 INFO:tasks.rados.rados.0.trial039.stdout:540: seq_num 258 ranges {736206=453891,1881413=454793} 2026-02-25T08:44:04.108 INFO:tasks.rados.rados.0.trial039.stdout:540: writing trial03942481-17 from 736206 to 1190097 tid 1 2026-02-25T08:44:04.113 INFO:tasks.rados.rados.0.trial039.stdout:540: writing trial03942481-17 from 1881413 to 2336206 tid 2 2026-02-25T08:44:04.113 INFO:tasks.rados.rados.0.trial039.stdout:541: write (excl) oid 10 current snap is 71 2026-02-25T08:44:04.113 INFO:tasks.rados.rados.0.trial039.stdout:541: seq_num 259 ranges {753013=645493,2056248=296765} 2026-02-25T08:44:04.118 INFO:tasks.rados.rados.0.trial039.stdout:541: writing trial03942481-10 from 753013 to 1398506 tid 1 2026-02-25T08:44:04.121 INFO:tasks.rados.rados.0.trial039.stdout:541: writing trial03942481-10 from 2056248 to 2353013 tid 2 2026-02-25T08:44:04.122 INFO:tasks.rados.rados.0.trial039.stdout:542: write (excl) oid 9 current snap is 71 2026-02-25T08:44:04.122 INFO:tasks.rados.rados.0.trial039.stdout:542: seq_num 260 ranges {769820=720742,2114730=255090} 2026-02-25T08:44:04.127 INFO:tasks.rados.rados.0.trial039.stdout:542: writing trial03942481-9 from 769820 to 1490562 tid 1 2026-02-25T08:44:04.130 INFO:tasks.rados.rados.0.trial039.stdout:542: writing trial03942481-9 from 2114730 to 2369820 tid 2 2026-02-25T08:44:04.131 INFO:tasks.rados.rados.0.trial039.stdout:543: write (excl) oid 43 current snap is 71 2026-02-25T08:44:04.131 INFO:tasks.rados.rados.0.trial039.stdout:543: seq_num 261 ranges {786627=795991,2173212=213415} 2026-02-25T08:44:04.136 INFO:tasks.rados.rados.0.trial039.stdout:543: writing trial03942481-43 from 786627 to 1582618 tid 1 2026-02-25T08:44:04.140 INFO:tasks.rados.rados.0.trial039.stdout:543: writing trial03942481-43 from 2173212 to 2386627 tid 2 2026-02-25T08:44:04.140 INFO:tasks.rados.rados.0.trial039.stdout:544: read oid 15 snap 68 2026-02-25T08:44:04.140 INFO:tasks.rados.rados.0.trial039.stdout:544: expect deleted 2026-02-25T08:44:04.140 INFO:tasks.rados.rados.0.trial039.stdout:545: write oid 18 current snap is 71 2026-02-25T08:44:04.140 INFO:tasks.rados.rados.0.trial039.stdout:545: seq_num 262 ranges {403434=471240,1315341=400756} 2026-02-25T08:44:04.143 INFO:tasks.rados.rados.0.trial039.stdout:545: writing trial03942481-18 from 403434 to 874674 tid 1 2026-02-25T08:44:04.146 INFO:tasks.rados.rados.0.trial039.stdout:545: writing trial03942481-18 from 1315341 to 1716097 tid 2 2026-02-25T08:44:04.147 INFO:tasks.rados.rados.0.trial039.stdout:546: read oid 36 snap -1 2026-02-25T08:44:04.147 INFO:tasks.rados.rados.0.trial039.stdout:546: expect (ObjNum 249 snap 70 seq_num 249) 2026-02-25T08:44:04.147 INFO:tasks.rados.rados.0.trial039.stdout:547: write oid 37 current snap is 71 2026-02-25T08:44:04.147 INFO:tasks.rados.rados.0.trial039.stdout:547: seq_num 263 ranges {420241=546489,1373823=544414} 2026-02-25T08:44:04.150 INFO:tasks.rados.rados.0.trial039.stdout:547: writing trial03942481-37 from 420241 to 966730 tid 1 2026-02-25T08:44:04.154 INFO:tasks.rados.rados.0.trial039.stdout:547: writing trial03942481-37 from 1373823 to 1918237 tid 2 2026-02-25T08:44:04.154 INFO:tasks.rados.rados.0.trial039.stdout:548: read oid 40 snap -1 2026-02-25T08:44:04.154 INFO:tasks.rados.rados.0.trial039.stdout:548: expect (ObjNum 235 snap 64 seq_num 235) 2026-02-25T08:44:04.154 INFO:tasks.rados.rados.0.trial039.stdout:549: delete oid 50 current snap is 71 2026-02-25T08:44:04.154 INFO:tasks.rados.rados.0.trial039.stdout:538: finishing write tid 1 to trial03942481-42 2026-02-25T08:44:04.154 INFO:tasks.rados.rados.0.trial039.stdout:538: finishing write tid 2 to trial03942481-42 2026-02-25T08:44:04.163 INFO:tasks.rados.rados.0.trial039.stdout:538: finishing write tid 3 to trial03942481-42 2026-02-25T08:44:04.164 INFO:tasks.rados.rados.0.trial039.stdout:540: finishing write tid 1 to trial03942481-17 2026-02-25T08:44:04.164 INFO:tasks.rados.rados.0.trial039.stdout:538: finishing write tid 4 to trial03942481-42 2026-02-25T08:44:04.164 INFO:tasks.rados.rados.0.trial039.stdout:538: oid 42 updating version 0 to 190 2026-02-25T08:44:04.164 INFO:tasks.rados.rados.0.trial039.stdout:538: oid 42 version 190 is already newer than 190 2026-02-25T08:44:04.164 INFO:tasks.rados.rados.0.trial039.stdout:538: oid 42 version 190 is already newer than 190 2026-02-25T08:44:04.165 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 42 v 190 (ObjNum 256 snap 71 seq_num 256) dirty exists 2026-02-25T08:44:04.165 INFO:tasks.rados.rados.0.trial039.stdout:538: left oid 42 (ObjNum 256 snap 71 seq_num 256) 2026-02-25T08:44:04.171 INFO:tasks.rados.rados.0.trial039.stdout:542: finishing write tid 1 to trial03942481-9 2026-02-25T08:44:04.180 INFO:tasks.rados.rados.0.trial039.stdout:537: finishing write tid 1 to trial03942481-41 2026-02-25T08:44:04.189 INFO:tasks.rados.rados.0.trial039.stdout:540: finishing write tid 2 to trial03942481-17 2026-02-25T08:44:04.190 INFO:tasks.rados.rados.0.trial039.stdout:545: finishing write tid 1 to trial03942481-18 2026-02-25T08:44:04.196 INFO:tasks.rados.rados.0.trial039.stdout:545: finishing write tid 2 to trial03942481-18 2026-02-25T08:44:04.197 INFO:tasks.rados.rados.0.trial039.stdout:536: expect (ObjNum 240 snap 65 seq_num 240) 2026-02-25T08:44:04.221 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1320577 2026-02-25T08:44:04.245 INFO:tasks.rados.rados.0.trial039.stdout:542: finishing write tid 2 to trial03942481-9 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:545: finishing write tid 3 to trial03942481-18 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:545: finishing write tid 4 to trial03942481-18 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:545: oid 18 updating version 0 to 84 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:545: oid 18 version 84 is already newer than 84 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:545: oid 18 version 84 is already newer than 84 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 18 v 84 (ObjNum 262 snap 71 seq_num 262) dirty exists 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:545: left oid 18 (ObjNum 262 snap 71 seq_num 262) 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:542: finishing write tid 3 to trial03942481-9 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:542: finishing write tid 4 to trial03942481-9 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:542: oid 9 updating version 0 to 24 2026-02-25T08:44:04.246 INFO:tasks.rados.rados.0.trial039.stdout:542: oid 9 version 24 is already newer than 22 2026-02-25T08:44:04.247 INFO:tasks.rados.rados.0.trial039.stdout:542: oid 9 version 24 is already newer than 24 2026-02-25T08:44:04.247 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 9 v 24 (ObjNum 260 snap 71 seq_num 260) dirty exists 2026-02-25T08:44:04.247 INFO:tasks.rados.rados.0.trial039.stdout:542: left oid 9 (ObjNum 260 snap 71 seq_num 260) 2026-02-25T08:44:04.247 INFO:tasks.rados.rados.0.trial039.stdout:548: expect (ObjNum 235 snap 64 seq_num 235) 2026-02-25T08:44:04.303 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:04.336 INFO:tasks.rados.rados.0.trial039.stdout:540: finishing write tid 3 to trial03942481-17 2026-02-25T08:44:04.337 INFO:tasks.rados.rados.0.trial039.stdout:540: finishing write tid 4 to trial03942481-17 2026-02-25T08:44:04.337 INFO:tasks.rados.rados.0.trial039.stdout:540: oid 17 updating version 0 to 99 2026-02-25T08:44:04.337 INFO:tasks.rados.rados.0.trial039.stdout:540: oid 17 updating version 99 to 101 2026-02-25T08:44:04.337 INFO:tasks.rados.rados.0.trial039.stdout:540: oid 17 version 101 is already newer than 101 2026-02-25T08:44:04.337 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 17 v 101 (ObjNum 258 snap 71 seq_num 258) dirty exists 2026-02-25T08:44:04.337 INFO:tasks.rados.rados.0.trial039.stdout:540: left oid 17 (ObjNum 258 snap 71 seq_num 258) 2026-02-25T08:44:04.337 INFO:tasks.rados.rados.0.trial039.stdout:546: expect (ObjNum 249 snap 70 seq_num 249) 2026-02-25T08:44:04.374 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:04.393 INFO:tasks.rados.rados.0.trial039.stdout:539: finishing write tid 1 to trial03942481-46 2026-02-25T08:44:04.393 INFO:tasks.rados.rados.0.trial039.stdout:547: finishing write tid 1 to trial03942481-37 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:536: done (14 left) 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:538: done (13 left) 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:540: done (12 left) 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:542: done (11 left) 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:544: done (10 left) 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:545: done (9 left) 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:546: done (8 left) 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:548: done (7 left) 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:549: done (6 left) 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:550: write oid 29 current snap is 71 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:547: finishing write tid 2 to trial03942481-37 2026-02-25T08:44:04.394 INFO:tasks.rados.rados.0.trial039.stdout:550: seq_num 265 ranges {453855=696987,1890787=548083} 2026-02-25T08:44:04.397 INFO:tasks.rados.rados.0.trial039.stdout:550: writing trial03942481-29 from 453855 to 1150842 tid 1 2026-02-25T08:44:04.404 INFO:tasks.rados.rados.0.trial039.stdout:550: writing trial03942481-29 from 1890787 to 2438870 tid 2 2026-02-25T08:44:04.404 INFO:tasks.rados.rados.0.trial039.stdout:551: delete oid 17 current snap is 71 2026-02-25T08:44:04.404 INFO:tasks.rados.rados.0.trial039.stdout:547: finishing write tid 3 to trial03942481-37 2026-02-25T08:44:04.404 INFO:tasks.rados.rados.0.trial039.stdout:547: finishing write tid 4 to trial03942481-37 2026-02-25T08:44:04.404 INFO:tasks.rados.rados.0.trial039.stdout:547: oid 37 updating version 0 to 104 2026-02-25T08:44:04.404 INFO:tasks.rados.rados.0.trial039.stdout:547: oid 37 version 104 is already newer than 104 2026-02-25T08:44:04.404 INFO:tasks.rados.rados.0.trial039.stdout:547: oid 37 version 104 is already newer than 104 2026-02-25T08:44:04.404 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 37 v 104 (ObjNum 263 snap 71 seq_num 263) dirty exists 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:547: left oid 37 (ObjNum 263 snap 71 seq_num 263) 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:537: finishing write tid 2 to trial03942481-41 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:537: finishing write tid 3 to trial03942481-41 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:537: finishing write tid 4 to trial03942481-41 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:537: oid 41 updating version 0 to 43 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:537: oid 41 version 43 is already newer than 43 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:537: oid 41 version 43 is already newer than 41 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 41 v 43 (ObjNum 255 snap 71 seq_num 255) dirty exists 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:537: left oid 41 (ObjNum 255 snap 71 seq_num 255) 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:539: finishing write tid 2 to trial03942481-46 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:535: finishing write tid 1 to trial03942481-26 2026-02-25T08:44:04.405 INFO:tasks.rados.rados.0.trial039.stdout:539: finishing write tid 3 to trial03942481-46 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:539: finishing write tid 4 to trial03942481-46 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:539: oid 46 updating version 0 to 63 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:539: oid 46 updating version 63 to 65 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:539: oid 46 version 65 is already newer than 65 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 46 v 65 (ObjNum 257 snap 71 seq_num 257) dirty exists 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:539: left oid 46 (ObjNum 257 snap 71 seq_num 257) 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:535: finishing write tid 2 to trial03942481-26 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:535: finishing write tid 3 to trial03942481-26 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:535: finishing write tid 4 to trial03942481-26 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:535: oid 26 updating version 0 to 50 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:535: oid 26 version 50 is already newer than 50 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:535: oid 26 version 50 is already newer than 50 2026-02-25T08:44:04.406 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 26 v 50 (ObjNum 254 snap 71 seq_num 254) dirty exists 2026-02-25T08:44:04.407 INFO:tasks.rados.rados.0.trial039.stdout:535: left oid 26 (ObjNum 254 snap 71 seq_num 254) 2026-02-25T08:44:04.407 INFO:tasks.rados.rados.0.trial039.stdout:541: finishing write tid 1 to trial03942481-10 2026-02-25T08:44:04.425 INFO:tasks.rados.rados.0.trial039.stdout:535: done (7 left) 2026-02-25T08:44:04.425 INFO:tasks.rados.rados.0.trial039.stdout:537: done (6 left) 2026-02-25T08:44:04.425 INFO:tasks.rados.rados.0.trial039.stdout:539: done (5 left) 2026-02-25T08:44:04.425 INFO:tasks.rados.rados.0.trial039.stdout:547: done (4 left) 2026-02-25T08:44:04.425 INFO:tasks.rados.rados.0.trial039.stdout:551: done (3 left) 2026-02-25T08:44:04.425 INFO:tasks.rados.rados.0.trial039.stdout:552: delete oid 3 current snap is 71 2026-02-25T08:44:04.427 INFO:tasks.rados.rados.0.trial039.stdout:552: done (3 left) 2026-02-25T08:44:04.427 INFO:tasks.rados.rados.0.trial039.stdout:553: delete oid 32 current snap is 71 2026-02-25T08:44:04.430 INFO:tasks.rados.rados.0.trial039.stdout:553: done (3 left) 2026-02-25T08:44:04.431 INFO:tasks.rados.rados.0.trial039.stdout:554: read oid 35 snap 66 2026-02-25T08:44:04.431 INFO:tasks.rados.rados.0.trial039.stdout:554: expect (ObjNum 139 snap 22 seq_num 139) 2026-02-25T08:44:04.431 INFO:tasks.rados.rados.0.trial039.stdout:555: snap_remove snap 57 2026-02-25T08:44:05.108 INFO:tasks.rados.rados.0.trial039.stdout:555: done (4 left) 2026-02-25T08:44:05.108 INFO:tasks.rados.rados.0.trial039.stdout:556: snap_remove snap 64 2026-02-25T08:44:05.116 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:05.194 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:05.191+0000 7f981f5b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:05.196 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:05.195+0000 7f981f5b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:05.222 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:05.238 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:06.051 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:06.111 INFO:tasks.rados.rados.0.trial039.stdout:556: done (4 left) 2026-02-25T08:44:06.112 INFO:tasks.rados.rados.0.trial039.stdout:557: snap_create 2026-02-25T08:44:06.112 INFO:tasks.rados.rados.0.trial039.stdout:550: finishing write tid 1 to trial03942481-29 2026-02-25T08:44:06.112 INFO:tasks.rados.rados.0.trial039.stdout:554: expect (ObjNum 139 snap 22 seq_num 139) 2026-02-25T08:44:06.112 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:06.123 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:06.127 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:06.131 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:06.151 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:06.171 INFO:tasks.rados.rados.0.trial039.stdout:550: finishing write tid 2 to trial03942481-29 2026-02-25T08:44:06.171 INFO:tasks.rados.rados.0.trial039.stdout:550: finishing write tid 3 to trial03942481-29 2026-02-25T08:44:06.171 INFO:tasks.rados.rados.0.trial039.stdout:550: finishing write tid 4 to trial03942481-29 2026-02-25T08:44:06.171 INFO:tasks.rados.rados.0.trial039.stdout:550: oid 29 updating version 0 to 54 2026-02-25T08:44:06.171 INFO:tasks.rados.rados.0.trial039.stdout:550: oid 29 version 54 is already newer than 54 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:550: oid 29 version 54 is already newer than 54 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 29 v 54 (ObjNum 265 snap 71 seq_num 265) dirty exists 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:550: left oid 29 (ObjNum 265 snap 71 seq_num 265) 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:541: finishing write tid 2 to trial03942481-10 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:541: finishing write tid 3 to trial03942481-10 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:541: finishing write tid 4 to trial03942481-10 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:541: oid 10 updating version 0 to 76 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:541: oid 10 version 76 is already newer than 76 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:541: oid 10 version 76 is already newer than 74 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 10 v 76 (ObjNum 259 snap 71 seq_num 259) dirty exists 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:541: left oid 10 (ObjNum 259 snap 71 seq_num 259) 2026-02-25T08:44:06.172 INFO:tasks.rados.rados.0.trial039.stdout:543: finishing write tid 1 to trial03942481-43 2026-02-25T08:44:06.173 INFO:tasks.rados.rados.0.trial039.stdout:543: finishing write tid 2 to trial03942481-43 2026-02-25T08:44:06.173 INFO:tasks.rados.rados.0.trial039.stdout:543: finishing write tid 3 to trial03942481-43 2026-02-25T08:44:06.173 INFO:tasks.rados.rados.0.trial039.stdout:543: finishing write tid 4 to trial03942481-43 2026-02-25T08:44:06.173 INFO:tasks.rados.rados.0.trial039.stdout:543: oid 43 updating version 0 to 78 2026-02-25T08:44:06.173 INFO:tasks.rados.rados.0.trial039.stdout:543: oid 43 updating version 78 to 80 2026-02-25T08:44:06.173 INFO:tasks.rados.rados.0.trial039.stdout:543: oid 43 version 80 is already newer than 80 2026-02-25T08:44:06.173 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 43 v 80 (ObjNum 261 snap 71 seq_num 261) dirty exists 2026-02-25T08:44:06.173 INFO:tasks.rados.rados.0.trial039.stdout:543: left oid 43 (ObjNum 261 snap 71 seq_num 261) 2026-02-25T08:44:07.117 INFO:tasks.rados.rados.0.trial039.stdout:541: done (4 left) 2026-02-25T08:44:07.118 INFO:tasks.rados.rados.0.trial039.stdout:543: done (3 left) 2026-02-25T08:44:07.118 INFO:tasks.rados.rados.0.trial039.stdout:550: done (2 left) 2026-02-25T08:44:07.118 INFO:tasks.rados.rados.0.trial039.stdout:554: done (1 left) 2026-02-25T08:44:07.118 INFO:tasks.rados.rados.0.trial039.stdout:557: done (0 left) 2026-02-25T08:44:07.118 INFO:tasks.rados.rados.0.trial039.stdout:558: snap_remove snap 65 2026-02-25T08:44:07.239 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:07.316 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:07.313+0000 7f2ad62b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:07.318 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:07.317+0000 7f2ad502e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:07.344 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:07.360 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:07.361 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:08.119 INFO:tasks.rados.rados.0.trial039.stdout:558: done (0 left) 2026-02-25T08:44:08.119 INFO:tasks.rados.rados.0.trial039.stdout:559: write oid 9 current snap is 72 2026-02-25T08:44:08.119 INFO:tasks.rados.rados.0.trial039.stdout:559: seq_num 269 ranges {521083=714336,1724715=555421} 2026-02-25T08:44:08.123 INFO:tasks.rados.rados.0.trial039.stdout:559: writing trial03942481-9 from 521083 to 1235419 tid 1 2026-02-25T08:44:08.128 INFO:tasks.rados.rados.0.trial039.stdout:559: writing trial03942481-9 from 1724715 to 2280136 tid 2 2026-02-25T08:44:08.128 INFO:tasks.rados.rados.0.trial039.stdout:560: read oid 33 snap -1 2026-02-25T08:44:08.128 INFO:tasks.rados.rados.0.trial039.stdout:560: expect deleted 2026-02-25T08:44:08.128 INFO:tasks.rados.rados.0.trial039.stdout:561: delete oid 41 current snap is 72 2026-02-25T08:44:08.158 INFO:tasks.rados.rados.0.trial039.stdout:560: done (2 left) 2026-02-25T08:44:08.158 INFO:tasks.rados.rados.0.trial039.stdout:561: done (1 left) 2026-02-25T08:44:08.158 INFO:tasks.rados.rados.0.trial039.stdout:562: write oid 41 current snap is 72 2026-02-25T08:44:08.159 INFO:tasks.rados.rados.0.trial039.stdout:562: seq_num 271 ranges {554697=464834,1725326=559090} 2026-02-25T08:44:08.163 INFO:tasks.rados.rados.0.trial039.stdout:562: writing trial03942481-41 from 554697 to 1019531 tid 1 2026-02-25T08:44:08.168 INFO:tasks.rados.rados.0.trial039.stdout:562: writing trial03942481-41 from 1725326 to 2284416 tid 2 2026-02-25T08:44:08.168 INFO:tasks.rados.rados.0.trial039.stdout:563: snap_create 2026-02-25T08:44:08.168 INFO:tasks.rados.rados.0.trial039.stdout:559: finishing write tid 1 to trial03942481-9 2026-02-25T08:44:08.175 INFO:tasks.rados.rados.0.trial039.stdout:559: finishing write tid 2 to trial03942481-9 2026-02-25T08:44:08.179 INFO:tasks.rados.rados.0.trial039.stdout:562: finishing write tid 1 to trial03942481-41 2026-02-25T08:44:08.182 INFO:tasks.rados.rados.0.trial039.stdout:559: finishing write tid 3 to trial03942481-9 2026-02-25T08:44:08.184 INFO:tasks.rados.rados.0.trial039.stdout:559: finishing write tid 4 to trial03942481-9 2026-02-25T08:44:08.184 INFO:tasks.rados.rados.0.trial039.stdout:559: oid 9 updating version 0 to 28 2026-02-25T08:44:08.184 INFO:tasks.rados.rados.0.trial039.stdout:559: oid 9 version 28 is already newer than 28 2026-02-25T08:44:08.184 INFO:tasks.rados.rados.0.trial039.stdout:559: oid 9 version 28 is already newer than 28 2026-02-25T08:44:08.184 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 9 v 28 (ObjNum 269 snap 72 seq_num 269) dirty exists 2026-02-25T08:44:08.184 INFO:tasks.rados.rados.0.trial039.stdout:559: left oid 9 (ObjNum 269 snap 72 seq_num 269) 2026-02-25T08:44:08.196 INFO:tasks.rados.rados.0.trial039.stdout:562: finishing write tid 2 to trial03942481-41 2026-02-25T08:44:08.202 INFO:tasks.rados.rados.0.trial039.stdout:562: finishing write tid 3 to trial03942481-41 2026-02-25T08:44:08.203 INFO:tasks.rados.rados.0.trial039.stdout:562: finishing write tid 4 to trial03942481-41 2026-02-25T08:44:08.204 INFO:tasks.rados.rados.0.trial039.stdout:562: oid 41 updating version 0 to 48 2026-02-25T08:44:08.204 INFO:tasks.rados.rados.0.trial039.stdout:562: oid 41 version 48 is already newer than 48 2026-02-25T08:44:08.204 INFO:tasks.rados.rados.0.trial039.stdout:562: oid 41 version 48 is already newer than 48 2026-02-25T08:44:08.204 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 41 v 48 (ObjNum 271 snap 72 seq_num 271) dirty exists 2026-02-25T08:44:08.204 INFO:tasks.rados.rados.0.trial039.stdout:562: left oid 41 (ObjNum 271 snap 72 seq_num 271) 2026-02-25T08:44:09.125 INFO:tasks.rados.rados.0.trial039.stdout:559: done (2 left) 2026-02-25T08:44:09.125 INFO:tasks.rados.rados.0.trial039.stdout:562: done (1 left) 2026-02-25T08:44:09.125 INFO:tasks.rados.rados.0.trial039.stdout:563: done (0 left) 2026-02-25T08:44:09.125 INFO:tasks.rados.rados.0.trial039.stdout:564: read oid 42 snap -1 2026-02-25T08:44:09.126 INFO:tasks.rados.rados.0.trial039.stdout:564: expect (ObjNum 256 snap 71 seq_num 256) 2026-02-25T08:44:09.126 INFO:tasks.rados.rados.0.trial039.stdout:565: snap_remove snap 70 2026-02-25T08:44:09.362 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:09.439 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:09.437+0000 7fd2be5c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:09.441 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:09.440+0000 7fd2bd338640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:09.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~154s 2026-02-25T08:44:09.466 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:09.482 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:09.483 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:10.128 INFO:tasks.rados.rados.0.trial039.stdout:565: done (1 left) 2026-02-25T08:44:10.128 INFO:tasks.rados.rados.0.trial039.stdout:566: delete oid 35 current snap is 73 2026-02-25T08:44:10.128 INFO:tasks.rados.rados.0.trial039.stdout:564: expect (ObjNum 256 snap 71 seq_num 256) 2026-02-25T08:44:10.128 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:10.167 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:10.186 INFO:tasks.rados.rados.0.trial039.stdout:564: done (1 left) 2026-02-25T08:44:10.186 INFO:tasks.rados.rados.0.trial039.stdout:566: done (0 left) 2026-02-25T08:44:10.187 INFO:tasks.rados.rados.0.trial039.stdout:567: delete oid 37 current snap is 73 2026-02-25T08:44:10.211 INFO:tasks.rados.rados.0.trial039.stdout:567: done (0 left) 2026-02-25T08:44:10.211 INFO:tasks.rados.rados.0.trial039.stdout:568: delete oid 15 current snap is 73 2026-02-25T08:44:10.213 INFO:tasks.rados.rados.0.trial039.stdout:568: done (0 left) 2026-02-25T08:44:10.214 INFO:tasks.rados.rados.0.trial039.stdout:569: read oid 44 snap -1 2026-02-25T08:44:10.214 INFO:tasks.rados.rados.0.trial039.stdout:569: expect (ObjNum 252 snap 70 seq_num 252) 2026-02-25T08:44:10.214 INFO:tasks.rados.rados.0.trial039.stdout:570: snap_create 2026-02-25T08:44:10.234 INFO:tasks.rados.rados.0.trial039.stdout:569: expect (ObjNum 252 snap 70 seq_num 252) 2026-02-25T08:44:10.316 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:11.128 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:11.133 INFO:tasks.rados.rados.0.trial039.stdout:569: done (1 left) 2026-02-25T08:44:11.133 INFO:tasks.rados.rados.0.trial039.stdout:570: done (0 left) 2026-02-25T08:44:11.133 INFO:tasks.rados.rados.0.trial039.stdout:571: delete oid 37 current snap is 74 2026-02-25T08:44:11.150 INFO:tasks.rados.rados.0.trial039.stdout:571: done (0 left) 2026-02-25T08:44:11.151 INFO:tasks.rados.rados.0.trial039.stdout:572: read oid 47 snap -1 2026-02-25T08:44:11.151 INFO:tasks.rados.rados.0.trial039.stdout:572: expect (ObjNum 221 snap 57 seq_num 221) 2026-02-25T08:44:11.151 INFO:tasks.rados.rados.0.trial039.stdout:573: delete oid 6 current snap is 74 2026-02-25T08:44:11.174 INFO:tasks.rados.rados.0.trial039.stdout:572: expect (ObjNum 221 snap 57 seq_num 221) 2026-02-25T08:44:11.201 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:11.204 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:11.204 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:11.226 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:11.331 INFO:tasks.rados.rados.0.trial039.stdout:572: done (1 left) 2026-02-25T08:44:11.331 INFO:tasks.rados.rados.0.trial039.stdout:573: done (0 left) 2026-02-25T08:44:11.331 INFO:tasks.rados.rados.0.trial039.stdout:574: snap_remove snap 71 2026-02-25T08:44:11.484 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:11.562 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:11.559+0000 7f48ca8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:11.564 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:11.563+0000 7f48ca8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:11.589 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:11.604 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:11.605 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:12.138 INFO:tasks.rados.rados.0.trial039.stdout:574: done (0 left) 2026-02-25T08:44:12.138 INFO:tasks.rados.rados.0.trial039.stdout:575: read oid 44 snap -1 2026-02-25T08:44:12.138 INFO:tasks.rados.rados.0.trial039.stdout:575: expect (ObjNum 252 snap 70 seq_num 252) 2026-02-25T08:44:12.138 INFO:tasks.rados.rados.0.trial039.stdout:576: snap_remove snap 47 2026-02-25T08:44:13.071 INFO:tasks.rados.rados.0.trial039.stdout:576: done (1 left) 2026-02-25T08:44:13.071 INFO:tasks.rados.rados.0.trial039.stdout:577: write (excl) oid 43 current snap is 74 2026-02-25T08:44:13.071 INFO:tasks.rados.rados.0.trial039.stdout:577: seq_num 277 ranges {655539=632681,2076218=570097} 2026-02-25T08:44:13.076 INFO:tasks.rados.rados.0.trial039.stdout:577: writing trial03942481-43 from 655539 to 1288220 tid 1 2026-02-25T08:44:13.082 INFO:tasks.rados.rados.0.trial039.stdout:577: writing trial03942481-43 from 2076218 to 2646315 tid 2 2026-02-25T08:44:13.083 INFO:tasks.rados.rados.0.trial039.stdout:578: write oid 4 current snap is 74 2026-02-25T08:44:13.083 INFO:tasks.rados.rados.0.trial039.stdout:578: seq_num 278 ranges {672346=707930,2134700=537646} 2026-02-25T08:44:13.088 INFO:tasks.rados.rados.0.trial039.stdout:578: writing trial03942481-4 from 672346 to 1380276 tid 1 2026-02-25T08:44:13.093 INFO:tasks.rados.rados.0.trial039.stdout:578: writing trial03942481-4 from 2134700 to 2672346 tid 2 2026-02-25T08:44:13.094 INFO:tasks.rados.rados.0.trial039.stdout:579: read oid 29 snap -1 2026-02-25T08:44:13.094 INFO:tasks.rados.rados.0.trial039.stdout:579: expect (ObjNum 265 snap 71 seq_num 265) 2026-02-25T08:44:13.094 INFO:tasks.rados.rados.0.trial039.stdout:580: delete oid 33 current snap is 74 2026-02-25T08:44:13.094 INFO:tasks.rados.rados.0.trial039.stdout:575: expect (ObjNum 252 snap 70 seq_num 252) 2026-02-25T08:44:13.178 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:13.220 INFO:tasks.rados.rados.0.trial039.stdout:575: done (4 left) 2026-02-25T08:44:13.220 INFO:tasks.rados.rados.0.trial039.stdout:580: done (3 left) 2026-02-25T08:44:13.220 INFO:tasks.rados.rados.0.trial039.stdout:581: read oid 10 snap -1 2026-02-25T08:44:13.220 INFO:tasks.rados.rados.0.trial039.stdout:581: expect (ObjNum 259 snap 71 seq_num 259) 2026-02-25T08:44:13.220 INFO:tasks.rados.rados.0.trial039.stdout:582: snap_create 2026-02-25T08:44:13.220 INFO:tasks.rados.rados.0.trial039.stdout:579: expect (ObjNum 265 snap 71 seq_num 265) 2026-02-25T08:44:13.309 INFO:tasks.rados.rados.0.trial039.stdout:578: finishing write tid 1 to trial03942481-4 2026-02-25T08:44:13.310 INFO:tasks.rados.rados.0.trial039.stdout:578: finishing write tid 2 to trial03942481-4 2026-02-25T08:44:13.310 INFO:tasks.rados.rados.0.trial039.stdout:578: finishing write tid 3 to trial03942481-4 2026-02-25T08:44:13.310 INFO:tasks.rados.rados.0.trial039.stdout:577: finishing write tid 1 to trial03942481-43 2026-02-25T08:44:13.310 INFO:tasks.rados.rados.0.trial039.stdout:578: finishing write tid 4 to trial03942481-4 2026-02-25T08:44:13.310 INFO:tasks.rados.rados.0.trial039.stdout:578: oid 4 updating version 0 to 88 2026-02-25T08:44:13.310 INFO:tasks.rados.rados.0.trial039.stdout:578: oid 4 version 88 is already newer than 88 2026-02-25T08:44:13.310 INFO:tasks.rados.rados.0.trial039.stdout:578: oid 4 version 88 is already newer than 88 2026-02-25T08:44:13.310 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 88 (ObjNum 278 snap 74 seq_num 278) dirty exists 2026-02-25T08:44:13.310 INFO:tasks.rados.rados.0.trial039.stdout:578: left oid 4 (ObjNum 278 snap 74 seq_num 278) 2026-02-25T08:44:13.310 INFO:tasks.rados.rados.0.trial039.stdout:577: finishing write tid 2 to trial03942481-43 2026-02-25T08:44:13.311 INFO:tasks.rados.rados.0.trial039.stdout:577: finishing write tid 3 to trial03942481-43 2026-02-25T08:44:13.311 INFO:tasks.rados.rados.0.trial039.stdout:577: finishing write tid 4 to trial03942481-43 2026-02-25T08:44:13.311 INFO:tasks.rados.rados.0.trial039.stdout:577: oid 43 updating version 0 to 84 2026-02-25T08:44:13.311 INFO:tasks.rados.rados.0.trial039.stdout:577: oid 43 version 84 is already newer than 82 2026-02-25T08:44:13.311 INFO:tasks.rados.rados.0.trial039.stdout:577: oid 43 version 84 is already newer than 84 2026-02-25T08:44:13.311 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 43 v 84 (ObjNum 277 snap 74 seq_num 277) dirty exists 2026-02-25T08:44:13.311 INFO:tasks.rados.rados.0.trial039.stdout:577: left oid 43 (ObjNum 277 snap 74 seq_num 277) 2026-02-25T08:44:13.311 INFO:tasks.rados.rados.0.trial039.stdout:581: expect (ObjNum 259 snap 71 seq_num 259) 2026-02-25T08:44:13.327 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:13.606 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:13.684 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:13.682+0000 7fd2e3851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:13.686 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:13.685+0000 7fd2e3851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:13.713 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:13.730 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:14.144 INFO:tasks.rados.rados.0.trial039.stdout:577: done (4 left) 2026-02-25T08:44:14.144 INFO:tasks.rados.rados.0.trial039.stdout:578: done (3 left) 2026-02-25T08:44:14.144 INFO:tasks.rados.rados.0.trial039.stdout:579: done (2 left) 2026-02-25T08:44:14.144 INFO:tasks.rados.rados.0.trial039.stdout:581: done (1 left) 2026-02-25T08:44:14.144 INFO:tasks.rados.rados.0.trial039.stdout:582: done (0 left) 2026-02-25T08:44:14.144 INFO:tasks.rados.rados.0.trial039.stdout:583: delete oid 26 current snap is 75 2026-02-25T08:44:14.189 INFO:tasks.rados.rados.0.trial039.stdout:583: done (0 left) 2026-02-25T08:44:14.190 INFO:tasks.rados.rados.0.trial039.stdout:584: write (excl) oid 12 current snap is 75 2026-02-25T08:44:14.190 INFO:tasks.rados.rados.0.trial039.stdout:584: seq_num 281 ranges {722767=533677,1793793=461082} 2026-02-25T08:44:14.194 INFO:tasks.rados.rados.0.trial039.stdout:584: writing trial03942481-12 from 722767 to 1256444 tid 1 2026-02-25T08:44:14.198 INFO:tasks.rados.rados.0.trial039.stdout:584: writing trial03942481-12 from 1793793 to 2254875 tid 2 2026-02-25T08:44:14.198 INFO:tasks.rados.rados.0.trial039.stdout:585: snap_create 2026-02-25T08:44:14.231 INFO:tasks.rados.rados.0.trial039.stdout:584: finishing write tid 1 to trial03942481-12 2026-02-25T08:44:14.246 INFO:tasks.rados.rados.0.trial039.stdout:584: finishing write tid 2 to trial03942481-12 2026-02-25T08:44:14.253 INFO:tasks.rados.rados.0.trial039.stdout:584: finishing write tid 3 to trial03942481-12 2026-02-25T08:44:14.255 INFO:tasks.rados.rados.0.trial039.stdout:584: finishing write tid 4 to trial03942481-12 2026-02-25T08:44:14.255 INFO:tasks.rados.rados.0.trial039.stdout:584: oid 12 updating version 0 to 45 2026-02-25T08:44:14.255 INFO:tasks.rados.rados.0.trial039.stdout:584: oid 12 version 45 is already newer than 43 2026-02-25T08:44:14.255 INFO:tasks.rados.rados.0.trial039.stdout:584: oid 12 version 45 is already newer than 45 2026-02-25T08:44:14.255 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 12 v 45 (ObjNum 281 snap 75 seq_num 281) dirty exists 2026-02-25T08:44:14.255 INFO:tasks.rados.rados.0.trial039.stdout:584: left oid 12 (ObjNum 281 snap 75 seq_num 281) 2026-02-25T08:44:14.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~160s 2026-02-25T08:44:15.149 INFO:tasks.rados.rados.0.trial039.stdout:584: done (1 left) 2026-02-25T08:44:15.150 INFO:tasks.rados.rados.0.trial039.stdout:585: done (0 left) 2026-02-25T08:44:15.150 INFO:tasks.rados.rados.0.trial039.stdout:586: delete oid 23 current snap is 76 2026-02-25T08:44:15.173 INFO:tasks.rados.rados.0.trial039.stdout:586: done (0 left) 2026-02-25T08:44:15.173 INFO:tasks.rados.rados.0.trial039.stdout:587: delete oid 27 current snap is 76 2026-02-25T08:44:15.200 INFO:tasks.rados.rados.0.trial039.stdout:587: done (0 left) 2026-02-25T08:44:15.201 INFO:tasks.rados.rados.0.trial039.stdout:588: delete oid 18 current snap is 76 2026-02-25T08:44:15.225 INFO:tasks.rados.rados.0.trial039.stdout:588: done (0 left) 2026-02-25T08:44:15.225 INFO:tasks.rados.rados.0.trial039.stdout:589: delete oid 39 current snap is 76 2026-02-25T08:44:15.254 INFO:tasks.rados.rados.0.trial039.stdout:589: done (0 left) 2026-02-25T08:44:15.254 INFO:tasks.rados.rados.0.trial039.stdout:590: snap_remove snap 74 2026-02-25T08:44:15.731 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:15.810 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:15.807+0000 7fafb441a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:15.812 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:15.811+0000 7fafb441a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:15.839 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:15.856 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:16.153 INFO:tasks.rados.rados.0.trial039.stdout:590: done (0 left) 2026-02-25T08:44:16.153 INFO:tasks.rados.rados.0.trial039.stdout:591: read oid 6 snap -1 2026-02-25T08:44:16.154 INFO:tasks.rados.rados.0.trial039.stdout:591: expect deleted 2026-02-25T08:44:16.154 INFO:tasks.rados.rados.0.trial039.stdout:592: read oid 35 snap -1 2026-02-25T08:44:16.154 INFO:tasks.rados.rados.0.trial039.stdout:592: expect deleted 2026-02-25T08:44:16.154 INFO:tasks.rados.rados.0.trial039.stdout:593: snap_remove snap 75 2026-02-25T08:44:16.205 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:16.278 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:16.281 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:17.159 INFO:tasks.rados.rados.0.trial039.stdout:593: done (2 left) 2026-02-25T08:44:17.159 INFO:tasks.rados.rados.0.trial039.stdout:594: write (excl) oid 38 current snap is 76 2026-02-25T08:44:17.160 INFO:tasks.rados.rados.0.trial039.stdout:594: seq_num 286 ranges {406802=626275,1686203=728431} 2026-02-25T08:44:17.164 INFO:tasks.rados.rados.0.trial039.stdout:594: writing trial03942481-38 from 406802 to 1033077 tid 1 2026-02-25T08:44:17.169 INFO:tasks.rados.rados.0.trial039.stdout:594: writing trial03942481-38 from 1686203 to 2414634 tid 2 2026-02-25T08:44:17.169 INFO:tasks.rados.rados.0.trial039.stdout:595: snap_remove snap 72 2026-02-25T08:44:17.857 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:17.934 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:17.932+0000 7f44b2be4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:17.937 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:17.936+0000 7f44b2be4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:17.964 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:17.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:18.163 INFO:tasks.rados.rados.0.trial039.stdout:595: done (3 left) 2026-02-25T08:44:18.163 INFO:tasks.rados.rados.0.trial039.stdout:596: snap_create 2026-02-25T08:44:18.163 INFO:tasks.rados.rados.0.trial039.stdout:594: finishing write tid 1 to trial03942481-38 2026-02-25T08:44:18.163 INFO:tasks.rados.rados.0.trial039.stdout:594: finishing write tid 2 to trial03942481-38 2026-02-25T08:44:18.163 INFO:tasks.rados.rados.0.trial039.stdout:594: finishing write tid 3 to trial03942481-38 2026-02-25T08:44:18.163 INFO:tasks.rados.rados.0.trial039.stdout:594: finishing write tid 4 to trial03942481-38 2026-02-25T08:44:18.163 INFO:tasks.rados.rados.0.trial039.stdout:594: oid 38 updating version 0 to 93 2026-02-25T08:44:18.163 INFO:tasks.rados.rados.0.trial039.stdout:594: oid 38 version 93 is already newer than 91 2026-02-25T08:44:18.163 INFO:tasks.rados.rados.0.trial039.stdout:594: oid 38 version 93 is already newer than 93 2026-02-25T08:44:18.164 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 38 v 93 (ObjNum 286 snap 76 seq_num 286) dirty exists 2026-02-25T08:44:18.164 INFO:tasks.rados.rados.0.trial039.stdout:594: left oid 38 (ObjNum 286 snap 76 seq_num 286) 2026-02-25T08:44:19.165 INFO:tasks.rados.rados.0.trial039.stdout:591: done (3 left) 2026-02-25T08:44:19.166 INFO:tasks.rados.rados.0.trial039.stdout:592: done (2 left) 2026-02-25T08:44:19.166 INFO:tasks.rados.rados.0.trial039.stdout:594: done (1 left) 2026-02-25T08:44:19.166 INFO:tasks.rados.rados.0.trial039.stdout:596: done (0 left) 2026-02-25T08:44:19.166 INFO:tasks.rados.rados.0.trial039.stdout:597: snap_create 2026-02-25T08:44:19.981 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:20.059 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:20.056+0000 7f803c9cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:20.062 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:20.061+0000 7f803c9cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:20.088 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:20.105 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:20.169 INFO:tasks.rados.rados.0.trial039.stdout:597: done (0 left) 2026-02-25T08:44:20.169 INFO:tasks.rados.rados.0.trial039.stdout:598: read oid 1 snap -1 2026-02-25T08:44:20.170 INFO:tasks.rados.rados.0.trial039.stdout:598: expect (ObjNum 250 snap 70 seq_num 250) 2026-02-25T08:44:20.170 INFO:tasks.rados.rados.0.trial039.stdout:599: write oid 34 current snap is 78 2026-02-25T08:44:20.170 INFO:tasks.rados.rados.0.trial039.stdout:599: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2026-02-25T08:44:20.173 INFO:tasks.rados.rados.0.trial039.stdout:599: writing trial03942481-34 from 423609 to 1125133 tid 1 2026-02-25T08:44:20.177 INFO:tasks.rados.rados.0.trial039.stdout:599: writing trial03942481-34 from 1628332 to 2100421 tid 2 2026-02-25T08:44:20.180 INFO:tasks.rados.rados.0.trial039.stdout:599: writing trial03942481-34 from 2648987 to 2823609 tid 3 2026-02-25T08:44:20.181 INFO:tasks.rados.rados.0.trial039.stdout:600: write (excl) oid 4 current snap is 78 2026-02-25T08:44:20.181 INFO:tasks.rados.rados.0.trial039.stdout:600: seq_num 288 ranges {440416=776773,1686814=732100} 2026-02-25T08:44:20.185 INFO:tasks.rados.rados.0.trial039.stdout:600: writing trial03942481-4 from 440416 to 1217189 tid 1 2026-02-25T08:44:20.190 INFO:tasks.rados.rados.0.trial039.stdout:600: writing trial03942481-4 from 1686814 to 2418914 tid 2 2026-02-25T08:44:20.190 INFO:tasks.rados.rados.0.trial039.stdout:601: read oid 2 snap -1 2026-02-25T08:44:20.190 INFO:tasks.rados.rados.0.trial039.stdout:601: expect deleted 2026-02-25T08:44:20.190 INFO:tasks.rados.rados.0.trial039.stdout:602: delete oid 28 current snap is 78 2026-02-25T08:44:20.216 INFO:tasks.rados.rados.0.trial039.stdout:600: finishing write tid 1 to trial03942481-4 2026-02-25T08:44:20.221 INFO:tasks.rados.rados.0.trial039.stdout:601: done (4 left) 2026-02-25T08:44:20.221 INFO:tasks.rados.rados.0.trial039.stdout:602: done (3 left) 2026-02-25T08:44:20.221 INFO:tasks.rados.rados.0.trial039.stdout:603: read oid 35 snap -1 2026-02-25T08:44:20.222 INFO:tasks.rados.rados.0.trial039.stdout:603: expect deleted 2026-02-25T08:44:20.222 INFO:tasks.rados.rados.0.trial039.stdout:604: write (excl) oid 37 current snap is 78 2026-02-25T08:44:20.222 INFO:tasks.rados.rados.0.trial039.stdout:604: seq_num 290 ranges {474030=643624,1520131=735769} 2026-02-25T08:44:20.226 INFO:tasks.rados.rados.0.trial039.stdout:604: writing trial03942481-37 from 474030 to 1117654 tid 1 2026-02-25T08:44:20.230 INFO:tasks.rados.rados.0.trial039.stdout:604: writing trial03942481-37 from 1520131 to 2255900 tid 2 2026-02-25T08:44:20.231 INFO:tasks.rados.rados.0.trial039.stdout:605: read oid 18 snap 76 2026-02-25T08:44:20.231 INFO:tasks.rados.rados.0.trial039.stdout:605: expect deleted 2026-02-25T08:44:20.231 INFO:tasks.rados.rados.0.trial039.stdout:606: write (excl) oid 38 current snap is 78 2026-02-25T08:44:20.231 INFO:tasks.rados.rados.0.trial039.stdout:598: expect (ObjNum 250 snap 70 seq_num 250) 2026-02-25T08:44:20.287 INFO:tasks.rados.rados.0.trial039.stdout:606: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2026-02-25T08:44:20.293 INFO:tasks.rados.rados.0.trial039.stdout:606: writing trial03942481-38 from 490837 to 1209710 tid 1 2026-02-25T08:44:20.298 INFO:tasks.rados.rados.0.trial039.stdout:606: writing trial03942481-38 from 1862260 to 2341687 tid 2 2026-02-25T08:44:20.301 INFO:tasks.rados.rados.0.trial039.stdout:606: writing trial03942481-38 from 2875032 to 2890837 tid 3 2026-02-25T08:44:20.302 INFO:tasks.rados.rados.0.trial039.stdout:598: done (6 left) 2026-02-25T08:44:20.302 INFO:tasks.rados.rados.0.trial039.stdout:603: done (5 left) 2026-02-25T08:44:20.302 INFO:tasks.rados.rados.0.trial039.stdout:607: write oid 32 current snap is 78 2026-02-25T08:44:20.302 INFO:tasks.rados.rados.0.trial039.stdout:600: finishing write tid 2 to trial03942481-4 2026-02-25T08:44:20.302 INFO:tasks.rados.rados.0.trial039.stdout:600: finishing write tid 3 to trial03942481-4 2026-02-25T08:44:20.302 INFO:tasks.rados.rados.0.trial039.stdout:607: seq_num 292 ranges {507644=794122,1920742=623085} 2026-02-25T08:44:20.307 INFO:tasks.rados.rados.0.trial039.stdout:607: writing trial03942481-32 from 507644 to 1301766 tid 1 2026-02-25T08:44:20.312 INFO:tasks.rados.rados.0.trial039.stdout:607: writing trial03942481-32 from 1920742 to 2543827 tid 2 2026-02-25T08:44:20.312 INFO:tasks.rados.rados.0.trial039.stdout:605: done (5 left) 2026-02-25T08:44:20.312 INFO:tasks.rados.rados.0.trial039.stdout:608: write oid 41 current snap is 78 2026-02-25T08:44:20.312 INFO:tasks.rados.rados.0.trial039.stdout:599: finishing write tid 1 to trial03942481-34 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:600: finishing write tid 4 to trial03942481-4 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:600: oid 4 updating version 0 to 95 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:600: oid 4 updating version 95 to 97 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:600: oid 4 version 97 is already newer than 97 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 97 (ObjNum 288 snap 78 seq_num 288) dirty exists 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:600: left oid 4 (ObjNum 288 snap 78 seq_num 288) 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:604: finishing write tid 1 to trial03942481-37 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:599: finishing write tid 2 to trial03942481-34 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:599: finishing write tid 3 to trial03942481-34 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:604: finishing write tid 2 to trial03942481-37 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:599: finishing write tid 4 to trial03942481-34 2026-02-25T08:44:20.313 INFO:tasks.rados.rados.0.trial039.stdout:604: finishing write tid 3 to trial03942481-37 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:599: finishing write tid 5 to trial03942481-34 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:599: oid 34 updating version 0 to 129 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:599: oid 34 version 129 is already newer than 129 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:599: oid 34 version 129 is already newer than 129 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:599: oid 34 version 129 is already newer than 129 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 129 (ObjNum 287 snap 78 seq_num 287) dirty exists 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:599: left oid 34 (ObjNum 287 snap 78 seq_num 287) 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:604: finishing write tid 4 to trial03942481-37 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:604: oid 37 updating version 0 to 108 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:604: oid 37 updating version 108 to 110 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:604: oid 37 version 110 is already newer than 110 2026-02-25T08:44:20.314 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 37 v 110 (ObjNum 290 snap 78 seq_num 290) dirty exists 2026-02-25T08:44:20.315 INFO:tasks.rados.rados.0.trial039.stdout:604: left oid 37 (ObjNum 290 snap 78 seq_num 290) 2026-02-25T08:44:20.315 INFO:tasks.rados.rados.0.trial039.stdout:608: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2026-02-25T08:44:20.315 INFO:tasks.rados.rados.0.trial039.stdout:608: writing trial03942481-41 from 524451 to 993822 tid 1 2026-02-25T08:44:20.319 INFO:tasks.rados.rados.0.trial039.stdout:608: writing trial03942481-41 from 1579224 to 2062320 tid 2 2026-02-25T08:44:20.323 INFO:tasks.rados.rados.0.trial039.stdout:608: writing trial03942481-41 from 2529878 to 2924451 tid 3 2026-02-25T08:44:20.323 INFO:tasks.rados.rados.0.trial039.stdout:599: done (5 left) 2026-02-25T08:44:20.323 INFO:tasks.rados.rados.0.trial039.stdout:600: done (4 left) 2026-02-25T08:44:20.323 INFO:tasks.rados.rados.0.trial039.stdout:604: done (3 left) 2026-02-25T08:44:20.324 INFO:tasks.rados.rados.0.trial039.stdout:609: read oid 29 snap -1 2026-02-25T08:44:20.324 INFO:tasks.rados.rados.0.trial039.stdout:609: expect (ObjNum 265 snap 71 seq_num 265) 2026-02-25T08:44:20.324 INFO:tasks.rados.rados.0.trial039.stdout:610: read oid 30 snap -1 2026-02-25T08:44:20.324 INFO:tasks.rados.rados.0.trial039.stdout:610: expect (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:44:20.324 INFO:tasks.rados.rados.0.trial039.stdout:611: read oid 4 snap -1 2026-02-25T08:44:20.324 INFO:tasks.rados.rados.0.trial039.stdout:611: expect (ObjNum 288 snap 78 seq_num 288) 2026-02-25T08:44:20.324 INFO:tasks.rados.rados.0.trial039.stdout:612: write oid 46 current snap is 78 2026-02-25T08:44:20.324 INFO:tasks.rados.rados.0.trial039.stdout:612: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2026-02-25T08:44:20.327 INFO:tasks.rados.rados.0.trial039.stdout:612: writing trial03942481-46 from 541258 to 1085878 tid 1 2026-02-25T08:44:20.331 INFO:tasks.rados.rados.0.trial039.stdout:612: writing trial03942481-46 from 1637706 to 2264460 tid 2 2026-02-25T08:44:20.334 INFO:tasks.rados.rados.0.trial039.stdout:612: writing trial03942481-46 from 2840948 to 2941258 tid 3 2026-02-25T08:44:20.334 INFO:tasks.rados.rados.0.trial039.stdout:613: write oid 22 current snap is 78 2026-02-25T08:44:20.334 INFO:tasks.rados.rados.0.trial039.stdout:606: finishing write tid 1 to trial03942481-38 2026-02-25T08:44:20.334 INFO:tasks.rados.rados.0.trial039.stdout:613: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2026-02-25T08:44:20.338 INFO:tasks.rados.rados.0.trial039.stdout:613: writing trial03942481-22 from 558065 to 1177934 tid 1 2026-02-25T08:44:20.342 INFO:tasks.rados.rados.0.trial039.stdout:613: writing trial03942481-22 from 1579835 to 2066600 tid 2 2026-02-25T08:44:20.345 INFO:tasks.rados.rados.0.trial039.stdout:613: writing trial03942481-22 from 2468371 to 2958065 tid 3 2026-02-25T08:44:20.346 INFO:tasks.rados.rados.0.trial039.stdout:614: read oid 34 snap -1 2026-02-25T08:44:20.346 INFO:tasks.rados.rados.0.trial039.stdout:614: expect (ObjNum 287 snap 78 seq_num 287) 2026-02-25T08:44:20.346 INFO:tasks.rados.rados.0.trial039.stdout:615: delete oid 12 current snap is 78 2026-02-25T08:44:20.346 INFO:tasks.rados.rados.0.trial039.stdout:607: finishing write tid 1 to trial03942481-32 2026-02-25T08:44:20.351 INFO:tasks.rados.rados.0.trial039.stdout:606: finishing write tid 2 to trial03942481-38 2026-02-25T08:44:20.365 INFO:tasks.rados.rados.0.trial039.stdout:607: finishing write tid 2 to trial03942481-32 2026-02-25T08:44:20.369 INFO:tasks.rados.rados.0.trial039.stdout:606: finishing write tid 3 to trial03942481-38 2026-02-25T08:44:20.376 INFO:tasks.rados.rados.0.trial039.stdout:611: expect (ObjNum 288 snap 78 seq_num 288) 2026-02-25T08:44:20.411 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1559486 2026-02-25T08:44:20.437 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:20.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~165s 2026-02-25T08:44:20.476 INFO:tasks.rados.rados.0.trial039.stdout:610: expect (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:44:20.522 INFO:tasks.rados.rados.0.trial039.stdout:612: finishing write tid 1 to trial03942481-46 2026-02-25T08:44:20.523 INFO:tasks.rados.rados.0.trial039.stdout:610: done (9 left) 2026-02-25T08:44:20.523 INFO:tasks.rados.rados.0.trial039.stdout:611: done (8 left) 2026-02-25T08:44:20.523 INFO:tasks.rados.rados.0.trial039.stdout:615: done (7 left) 2026-02-25T08:44:20.523 INFO:tasks.rados.rados.0.trial039.stdout:616: read oid 30 snap -1 2026-02-25T08:44:20.523 INFO:tasks.rados.rados.0.trial039.stdout:616: expect (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:44:20.523 INFO:tasks.rados.rados.0.trial039.stdout:617: read oid 4 snap -1 2026-02-25T08:44:20.523 INFO:tasks.rados.rados.0.trial039.stdout:617: expect (ObjNum 288 snap 78 seq_num 288) 2026-02-25T08:44:20.523 INFO:tasks.rados.rados.0.trial039.stdout:618: snap_create 2026-02-25T08:44:20.523 INFO:tasks.rados.rados.0.trial039.stdout:606: finishing write tid 4 to trial03942481-38 2026-02-25T08:44:20.523 INFO:tasks.rados.rados.0.trial039.stdout:606: finishing write tid 5 to trial03942481-38 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:606: oid 38 updating version 0 to 102 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:606: oid 38 version 102 is already newer than 102 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:606: oid 38 version 102 is already newer than 102 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:606: oid 38 version 102 is already newer than 99 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 38 v 102 (ObjNum 291 snap 78 seq_num 291) dirty exists 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:606: left oid 38 (ObjNum 291 snap 78 seq_num 291) 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:607: finishing write tid 3 to trial03942481-32 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:607: finishing write tid 4 to trial03942481-32 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:607: oid 32 updating version 0 to 73 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:607: oid 32 version 73 is already newer than 73 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:607: oid 32 version 73 is already newer than 73 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 32 v 73 (ObjNum 292 snap 78 seq_num 292) dirty exists 2026-02-25T08:44:20.524 INFO:tasks.rados.rados.0.trial039.stdout:607: left oid 32 (ObjNum 292 snap 78 seq_num 292) 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:612: finishing write tid 2 to trial03942481-46 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:612: finishing write tid 3 to trial03942481-46 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:612: finishing write tid 4 to trial03942481-46 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:612: finishing write tid 5 to trial03942481-46 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:612: oid 46 updating version 0 to 75 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:612: oid 46 version 75 is already newer than 75 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:612: oid 46 version 75 is already newer than 75 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:612: oid 46 version 75 is already newer than 75 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 46 v 75 (ObjNum 294 snap 78 seq_num 294) dirty exists 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:612: left oid 46 (ObjNum 294 snap 78 seq_num 294) 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:613: finishing write tid 1 to trial03942481-22 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:613: finishing write tid 2 to trial03942481-22 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:608: finishing write tid 1 to trial03942481-41 2026-02-25T08:44:20.525 INFO:tasks.rados.rados.0.trial039.stdout:613: finishing write tid 3 to trial03942481-22 2026-02-25T08:44:20.526 INFO:tasks.rados.rados.0.trial039.stdout:609: expect (ObjNum 265 snap 71 seq_num 265) 2026-02-25T08:44:20.555 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:20.623 INFO:tasks.rados.rados.0.trial039.stdout:613: finishing write tid 4 to trial03942481-22 2026-02-25T08:44:20.624 INFO:tasks.rados.rados.0.trial039.stdout:613: finishing write tid 5 to trial03942481-22 2026-02-25T08:44:20.624 INFO:tasks.rados.rados.0.trial039.stdout:613: oid 22 updating version 0 to 34 2026-02-25T08:44:20.624 INFO:tasks.rados.rados.0.trial039.stdout:613: oid 22 version 34 is already newer than 34 2026-02-25T08:44:20.624 INFO:tasks.rados.rados.0.trial039.stdout:613: oid 22 version 34 is already newer than 34 2026-02-25T08:44:20.624 INFO:tasks.rados.rados.0.trial039.stdout:613: oid 22 version 34 is already newer than 34 2026-02-25T08:44:20.624 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 22 v 34 (ObjNum 295 snap 78 seq_num 295) dirty exists 2026-02-25T08:44:20.624 INFO:tasks.rados.rados.0.trial039.stdout:613: left oid 22 (ObjNum 295 snap 78 seq_num 295) 2026-02-25T08:44:20.624 INFO:tasks.rados.rados.0.trial039.stdout:608: finishing write tid 2 to trial03942481-41 2026-02-25T08:44:20.624 INFO:tasks.rados.rados.0.trial039.stdout:614: expect (ObjNum 287 snap 78 seq_num 287) 2026-02-25T08:44:20.733 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1027917 2026-02-25T08:44:20.781 INFO:tasks.rados.rados.0.trial039.stdout:608: finishing write tid 3 to trial03942481-41 2026-02-25T08:44:20.782 INFO:tasks.rados.rados.0.trial039.stdout:608: finishing write tid 4 to trial03942481-41 2026-02-25T08:44:20.782 INFO:tasks.rados.rados.0.trial039.stdout:608: finishing write tid 5 to trial03942481-41 2026-02-25T08:44:20.782 INFO:tasks.rados.rados.0.trial039.stdout:608: oid 41 updating version 0 to 53 2026-02-25T08:44:20.782 INFO:tasks.rados.rados.0.trial039.stdout:608: oid 41 version 53 is already newer than 53 2026-02-25T08:44:20.782 INFO:tasks.rados.rados.0.trial039.stdout:608: oid 41 version 53 is already newer than 53 2026-02-25T08:44:20.782 INFO:tasks.rados.rados.0.trial039.stdout:608: oid 41 version 53 is already newer than 53 2026-02-25T08:44:20.782 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 41 v 53 (ObjNum 293 snap 78 seq_num 293) dirty exists 2026-02-25T08:44:20.782 INFO:tasks.rados.rados.0.trial039.stdout:608: left oid 41 (ObjNum 293 snap 78 seq_num 293) 2026-02-25T08:44:20.782 INFO:tasks.rados.rados.0.trial039.stdout:616: expect (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:44:20.787 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2024860 2026-02-25T08:44:20.815 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:20.836 INFO:tasks.rados.rados.0.trial039.stdout:617: expect (ObjNum 288 snap 78 seq_num 288) 2026-02-25T08:44:20.840 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 880440 2026-02-25T08:44:20.872 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:20.912 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:21.175 INFO:tasks.rados.rados.0.trial039.stdout:606: done (9 left) 2026-02-25T08:44:21.175 INFO:tasks.rados.rados.0.trial039.stdout:607: done (8 left) 2026-02-25T08:44:21.175 INFO:tasks.rados.rados.0.trial039.stdout:608: done (7 left) 2026-02-25T08:44:21.175 INFO:tasks.rados.rados.0.trial039.stdout:609: done (6 left) 2026-02-25T08:44:21.175 INFO:tasks.rados.rados.0.trial039.stdout:612: done (5 left) 2026-02-25T08:44:21.176 INFO:tasks.rados.rados.0.trial039.stdout:613: done (4 left) 2026-02-25T08:44:21.176 INFO:tasks.rados.rados.0.trial039.stdout:614: done (3 left) 2026-02-25T08:44:21.176 INFO:tasks.rados.rados.0.trial039.stdout:616: done (2 left) 2026-02-25T08:44:21.176 INFO:tasks.rados.rados.0.trial039.stdout:617: done (1 left) 2026-02-25T08:44:21.176 INFO:tasks.rados.rados.0.trial039.stdout:618: done (0 left) 2026-02-25T08:44:21.176 INFO:tasks.rados.rados.0.trial039.stdout:619: snap_remove snap 77 2026-02-25T08:44:21.282 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:21.354 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:21.357 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:21.357 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:22.106 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:22.176 INFO:tasks.rados.rados.0.trial039.stdout:619: done (0 left) 2026-02-25T08:44:22.176 INFO:tasks.rados.rados.0.trial039.stdout:620: delete oid 41 current snap is 79 2026-02-25T08:44:22.188 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:22.185+0000 7f1f3f50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:22.191 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:22.190+0000 7f1f3f50b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:22.223 INFO:tasks.rados.rados.0.trial039.stdout:620: done (0 left) 2026-02-25T08:44:22.223 INFO:tasks.rados.rados.0.trial039.stdout:621: delete oid 50 current snap is 79 2026-02-25T08:44:22.226 INFO:tasks.rados.rados.0.trial039.stdout:621: done (0 left) 2026-02-25T08:44:22.226 INFO:tasks.rados.rados.0.trial039.stdout:622: snap_create 2026-02-25T08:44:22.231 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:22.247 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:23.073 INFO:tasks.rados.rados.0.trial039.stdout:622: done (0 left) 2026-02-25T08:44:23.073 INFO:tasks.rados.rados.0.trial039.stdout:623: read oid 6 snap -1 2026-02-25T08:44:23.073 INFO:tasks.rados.rados.0.trial039.stdout:623: expect deleted 2026-02-25T08:44:23.073 INFO:tasks.rados.rados.0.trial039.stdout:624: snap_remove snap 68 2026-02-25T08:44:24.074 INFO:tasks.rados.rados.0.trial039.stdout:624: done (1 left) 2026-02-25T08:44:24.074 INFO:tasks.rados.rados.0.trial039.stdout:625: write oid 23 current snap is 80 2026-02-25T08:44:24.074 INFO:tasks.rados.rados.0.trial039.stdout:625: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2026-02-25T08:44:24.079 INFO:tasks.rados.rados.0.trial039.stdout:625: writing trial03942481-23 from 625293 to 1262511 tid 1 2026-02-25T08:44:24.084 INFO:tasks.rados.rados.0.trial039.stdout:625: writing trial03942481-23 from 1813763 to 2307866 tid 2 2026-02-25T08:44:24.087 INFO:tasks.rados.rados.0.trial039.stdout:625: writing trial03942481-23 from 2978063 to 3025293 tid 3 2026-02-25T08:44:24.087 INFO:tasks.rados.rados.0.trial039.stdout:626: snap_create 2026-02-25T08:44:24.118 INFO:tasks.rados.rados.0.trial039.stdout:625: finishing write tid 1 to trial03942481-23 2026-02-25T08:44:24.124 INFO:tasks.rados.rados.0.trial039.stdout:625: finishing write tid 2 to trial03942481-23 2026-02-25T08:44:24.130 INFO:tasks.rados.rados.0.trial039.stdout:625: finishing write tid 3 to trial03942481-23 2026-02-25T08:44:24.134 INFO:tasks.rados.rados.0.trial039.stdout:625: finishing write tid 4 to trial03942481-23 2026-02-25T08:44:24.136 INFO:tasks.rados.rados.0.trial039.stdout:625: finishing write tid 5 to trial03942481-23 2026-02-25T08:44:24.136 INFO:tasks.rados.rados.0.trial039.stdout:625: oid 23 updating version 0 to 81 2026-02-25T08:44:24.136 INFO:tasks.rados.rados.0.trial039.stdout:625: oid 23 version 81 is already newer than 81 2026-02-25T08:44:24.136 INFO:tasks.rados.rados.0.trial039.stdout:625: oid 23 version 81 is already newer than 81 2026-02-25T08:44:24.136 INFO:tasks.rados.rados.0.trial039.stdout:625: oid 23 version 81 is already newer than 81 2026-02-25T08:44:24.136 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 23 v 81 (ObjNum 299 snap 80 seq_num 299) dirty exists 2026-02-25T08:44:24.136 INFO:tasks.rados.rados.0.trial039.stdout:625: left oid 23 (ObjNum 299 snap 80 seq_num 299) 2026-02-25T08:44:24.247 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:24.324 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:24.321+0000 7f9aa068b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:24.326 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:24.325+0000 7f9aa068b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:24.352 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:24.370 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:25.076 INFO:tasks.rados.rados.0.trial039.stdout:623: done (2 left) 2026-02-25T08:44:25.076 INFO:tasks.rados.rados.0.trial039.stdout:625: done (1 left) 2026-02-25T08:44:25.077 INFO:tasks.rados.rados.0.trial039.stdout:626: done (0 left) 2026-02-25T08:44:25.077 INFO:tasks.rados.rados.0.trial039.stdout:627: read oid 11 snap -1 2026-02-25T08:44:25.077 INFO:tasks.rados.rados.0.trial039.stdout:627: expect (ObjNum 241 snap 65 seq_num 241) 2026-02-25T08:44:25.077 INFO:tasks.rados.rados.0.trial039.stdout:628: read oid 24 snap 76 2026-02-25T08:44:25.077 INFO:tasks.rados.rados.0.trial039.stdout:628: expect (ObjNum 196 snap 49 seq_num 196) 2026-02-25T08:44:25.077 INFO:tasks.rados.rados.0.trial039.stdout:629: write oid 27 current snap is 81 2026-02-25T08:44:25.077 INFO:tasks.rados.rados.0.trial039.stdout:629: seq_num 300 ranges {642100=712467,1872245=637761} 2026-02-25T08:44:25.081 INFO:tasks.rados.rados.0.trial039.stdout:629: writing trial03942481-27 from 642100 to 1354567 tid 1 2026-02-25T08:44:25.086 INFO:tasks.rados.rados.0.trial039.stdout:629: writing trial03942481-27 from 1872245 to 2510006 tid 2 2026-02-25T08:44:25.086 INFO:tasks.rados.rados.0.trial039.stdout:630: write (excl) oid 36 current snap is 81 2026-02-25T08:44:25.086 INFO:tasks.rados.rados.0.trial039.stdout:630: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2026-02-25T08:44:25.092 INFO:tasks.rados.rados.0.trial039.stdout:630: writing trial03942481-36 from 658907 to 1446623 tid 1 2026-02-25T08:44:25.095 INFO:tasks.rados.rados.0.trial039.stdout:630: writing trial03942481-36 from 1930727 to 2428499 tid 2 2026-02-25T08:44:25.098 INFO:tasks.rados.rados.0.trial039.stdout:630: writing trial03942481-36 from 3032909 to 3058907 tid 3 2026-02-25T08:44:25.098 INFO:tasks.rados.rados.0.trial039.stdout:631: write oid 6 current snap is 81 2026-02-25T08:44:25.098 INFO:tasks.rados.rados.0.trial039.stdout:631: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2026-02-25T08:44:25.102 INFO:tasks.rados.rados.0.trial039.stdout:631: writing trial03942481-6 from 675714 to 1138679 tid 1 2026-02-25T08:44:25.106 INFO:tasks.rados.rados.0.trial039.stdout:631: writing trial03942481-6 from 1589209 to 2230639 tid 2 2026-02-25T08:44:25.110 INFO:tasks.rados.rados.0.trial039.stdout:631: writing trial03942481-6 from 2943979 to 3075714 tid 3 2026-02-25T08:44:25.110 INFO:tasks.rados.rados.0.trial039.stdout:632: delete oid 38 current snap is 81 2026-02-25T08:44:25.113 INFO:tasks.rados.rados.0.trial039.stdout:627: expect (ObjNum 241 snap 65 seq_num 241) 2026-02-25T08:44:25.182 INFO:tasks.rados.rados.0.trial039.stdout:628: expect (ObjNum 196 snap 49 seq_num 196) 2026-02-25T08:44:25.182 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:25.282 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:25.336 INFO:tasks.rados.rados.0.trial039.stdout:629: finishing write tid 1 to trial03942481-27 2026-02-25T08:44:25.337 INFO:tasks.rados.rados.0.trial039.stdout:627: done (5 left) 2026-02-25T08:44:25.337 INFO:tasks.rados.rados.0.trial039.stdout:628: done (4 left) 2026-02-25T08:44:25.337 INFO:tasks.rados.rados.0.trial039.stdout:632: done (3 left) 2026-02-25T08:44:25.337 INFO:tasks.rados.rados.0.trial039.stdout:633: write oid 47 current snap is 81 2026-02-25T08:44:25.337 INFO:tasks.rados.rados.0.trial039.stdout:629: finishing write tid 2 to trial03942481-27 2026-02-25T08:44:25.337 INFO:tasks.rados.rados.0.trial039.stdout:633: seq_num 304 ranges {709328=613463,1989820=645099} 2026-02-25T08:44:25.341 INFO:tasks.rados.rados.0.trial039.stdout:633: writing trial03942481-47 from 709328 to 1322791 tid 1 2026-02-25T08:44:25.346 INFO:tasks.rados.rados.0.trial039.stdout:633: writing trial03942481-47 from 1989820 to 2634919 tid 2 2026-02-25T08:44:25.346 INFO:tasks.rados.rados.0.trial039.stdout:634: write (excl) oid 20 current snap is 81 2026-02-25T08:44:25.347 INFO:tasks.rados.rados.0.trial039.stdout:634: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2026-02-25T08:44:25.350 INFO:tasks.rados.rados.0.trial039.stdout:634: writing trial03942481-20 from 726135 to 1131200 tid 1 2026-02-25T08:44:25.356 INFO:tasks.rados.rados.0.trial039.stdout:634: writing trial03942481-20 from 1764655 to 2553412 tid 2 2026-02-25T08:44:25.359 INFO:tasks.rados.rados.0.trial039.stdout:634: writing trial03942481-20 from 3026248 to 3126135 tid 3 2026-02-25T08:44:25.359 INFO:tasks.rados.rados.0.trial039.stdout:635: read oid 5 snap -1 2026-02-25T08:44:25.359 INFO:tasks.rados.rados.0.trial039.stdout:635: expect (ObjNum 152 snap 29 seq_num 152) 2026-02-25T08:44:25.359 INFO:tasks.rados.rados.0.trial039.stdout:636: read oid 14 snap -1 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:636: expect deleted 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:637: snap_create 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:629: finishing write tid 3 to trial03942481-27 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:629: finishing write tid 4 to trial03942481-27 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:629: oid 27 updating version 0 to 76 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:629: oid 27 version 76 is already newer than 76 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:629: oid 27 version 76 is already newer than 76 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 27 v 76 (ObjNum 300 snap 81 seq_num 300) dirty exists 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:629: left oid 27 (ObjNum 300 snap 81 seq_num 300) 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:630: finishing write tid 1 to trial03942481-36 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:630: finishing write tid 2 to trial03942481-36 2026-02-25T08:44:25.360 INFO:tasks.rados.rados.0.trial039.stdout:630: finishing write tid 3 to trial03942481-36 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:630: finishing write tid 4 to trial03942481-36 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:630: finishing write tid 5 to trial03942481-36 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:630: oid 36 updating version 0 to 78 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:630: oid 36 updating version 78 to 81 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:630: oid 36 version 81 is already newer than 81 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:630: oid 36 version 81 is already newer than 81 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 36 v 81 (ObjNum 301 snap 81 seq_num 301) dirty exists 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:630: left oid 36 (ObjNum 301 snap 81 seq_num 301) 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:631: finishing write tid 1 to trial03942481-6 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:631: finishing write tid 2 to trial03942481-6 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:631: finishing write tid 3 to trial03942481-6 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:631: finishing write tid 4 to trial03942481-6 2026-02-25T08:44:25.361 INFO:tasks.rados.rados.0.trial039.stdout:631: finishing write tid 5 to trial03942481-6 2026-02-25T08:44:25.362 INFO:tasks.rados.rados.0.trial039.stdout:631: oid 6 updating version 0 to 232 2026-02-25T08:44:25.362 INFO:tasks.rados.rados.0.trial039.stdout:631: oid 6 version 232 is already newer than 232 2026-02-25T08:44:25.362 INFO:tasks.rados.rados.0.trial039.stdout:631: oid 6 version 232 is already newer than 232 2026-02-25T08:44:25.362 INFO:tasks.rados.rados.0.trial039.stdout:631: oid 6 version 232 is already newer than 232 2026-02-25T08:44:25.362 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 6 v 232 (ObjNum 302 snap 81 seq_num 302) dirty exists 2026-02-25T08:44:25.362 INFO:tasks.rados.rados.0.trial039.stdout:631: left oid 6 (ObjNum 302 snap 81 seq_num 302) 2026-02-25T08:44:25.369 INFO:tasks.rados.rados.0.trial039.stdout:635: expect (ObjNum 152 snap 29 seq_num 152) 2026-02-25T08:44:25.369 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 34996 2026-02-25T08:44:25.402 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1423370 2026-02-25T08:44:25.416 INFO:tasks.rados.rados.0.trial039.stdout:633: finishing write tid 1 to trial03942481-47 2026-02-25T08:44:25.416 INFO:tasks.rados.rados.0.trial039.stdout:634: finishing write tid 1 to trial03942481-20 2026-02-25T08:44:25.416 INFO:tasks.rados.rados.0.trial039.stdout:633: finishing write tid 2 to trial03942481-47 2026-02-25T08:44:25.416 INFO:tasks.rados.rados.0.trial039.stdout:633: finishing write tid 3 to trial03942481-47 2026-02-25T08:44:25.416 INFO:tasks.rados.rados.0.trial039.stdout:634: finishing write tid 2 to trial03942481-20 2026-02-25T08:44:25.416 INFO:tasks.rados.rados.0.trial039.stdout:633: finishing write tid 4 to trial03942481-47 2026-02-25T08:44:25.416 INFO:tasks.rados.rados.0.trial039.stdout:633: oid 47 updating version 0 to 66 2026-02-25T08:44:25.416 INFO:tasks.rados.rados.0.trial039.stdout:633: oid 47 version 66 is already newer than 66 2026-02-25T08:44:25.416 INFO:tasks.rados.rados.0.trial039.stdout:633: oid 47 version 66 is already newer than 66 2026-02-25T08:44:25.416 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 47 v 66 (ObjNum 304 snap 81 seq_num 304) dirty exists 2026-02-25T08:44:25.417 INFO:tasks.rados.rados.0.trial039.stdout:633: left oid 47 (ObjNum 304 snap 81 seq_num 304) 2026-02-25T08:44:25.417 INFO:tasks.rados.rados.0.trial039.stdout:634: finishing write tid 3 to trial03942481-20 2026-02-25T08:44:25.417 INFO:tasks.rados.rados.0.trial039.stdout:634: finishing write tid 4 to trial03942481-20 2026-02-25T08:44:25.417 INFO:tasks.rados.rados.0.trial039.stdout:634: finishing write tid 5 to trial03942481-20 2026-02-25T08:44:25.418 INFO:tasks.rados.rados.0.trial039.stdout:634: oid 20 updating version 0 to 134 2026-02-25T08:44:25.418 INFO:tasks.rados.rados.0.trial039.stdout:634: oid 20 version 134 is already newer than 134 2026-02-25T08:44:25.418 INFO:tasks.rados.rados.0.trial039.stdout:634: oid 20 version 134 is already newer than 131 2026-02-25T08:44:25.418 INFO:tasks.rados.rados.0.trial039.stdout:634: oid 20 version 134 is already newer than 134 2026-02-25T08:44:25.418 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 20 v 134 (ObjNum 305 snap 81 seq_num 305) dirty exists 2026-02-25T08:44:25.418 INFO:tasks.rados.rados.0.trial039.stdout:634: left oid 20 (ObjNum 305 snap 81 seq_num 305) 2026-02-25T08:44:25.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~171s 2026-02-25T08:44:26.080 INFO:tasks.rados.rados.0.trial039.stdout:629: done (7 left) 2026-02-25T08:44:26.080 INFO:tasks.rados.rados.0.trial039.stdout:630: done (6 left) 2026-02-25T08:44:26.080 INFO:tasks.rados.rados.0.trial039.stdout:631: done (5 left) 2026-02-25T08:44:26.080 INFO:tasks.rados.rados.0.trial039.stdout:633: done (4 left) 2026-02-25T08:44:26.080 INFO:tasks.rados.rados.0.trial039.stdout:634: done (3 left) 2026-02-25T08:44:26.080 INFO:tasks.rados.rados.0.trial039.stdout:635: done (2 left) 2026-02-25T08:44:26.080 INFO:tasks.rados.rados.0.trial039.stdout:636: done (1 left) 2026-02-25T08:44:26.080 INFO:tasks.rados.rados.0.trial039.stdout:637: done (0 left) 2026-02-25T08:44:26.081 INFO:tasks.rados.rados.0.trial039.stdout:638: write oid 13 current snap is 82 2026-02-25T08:44:26.081 INFO:tasks.rados.rados.0.trial039.stdout:638: seq_num 306 ranges {742942=480314,1823137=648768} 2026-02-25T08:44:26.084 INFO:tasks.rados.rados.0.trial039.stdout:638: writing trial03942481-13 from 742942 to 1223256 tid 1 2026-02-25T08:44:26.089 INFO:tasks.rados.rados.0.trial039.stdout:638: writing trial03942481-13 from 1823137 to 2471905 tid 2 2026-02-25T08:44:26.089 INFO:tasks.rados.rados.0.trial039.stdout:639: write (excl) oid 46 current snap is 82 2026-02-25T08:44:26.089 INFO:tasks.rados.rados.0.trial039.stdout:639: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2026-02-25T08:44:26.094 INFO:tasks.rados.rados.0.trial039.stdout:639: writing trial03942481-46 from 759749 to 1315312 tid 1 2026-02-25T08:44:26.099 INFO:tasks.rados.rados.0.trial039.stdout:639: writing trial03942481-46 from 1765266 to 2557692 tid 2 2026-02-25T08:44:26.101 INFO:tasks.rados.rados.0.trial039.stdout:639: writing trial03942481-46 from 2964741 to 3159749 tid 3 2026-02-25T08:44:26.101 INFO:tasks.rados.rados.0.trial039.stdout:640: delete oid 45 current snap is 82 2026-02-25T08:44:26.114 INFO:tasks.rados.rados.0.trial039.stdout:638: finishing write tid 1 to trial03942481-13 2026-02-25T08:44:26.126 INFO:tasks.rados.rados.0.trial039.stdout:638: finishing write tid 2 to trial03942481-13 2026-02-25T08:44:26.131 INFO:tasks.rados.rados.0.trial039.stdout:638: finishing write tid 3 to trial03942481-13 2026-02-25T08:44:26.132 INFO:tasks.rados.rados.0.trial039.stdout:639: finishing write tid 1 to trial03942481-46 2026-02-25T08:44:26.134 INFO:tasks.rados.rados.0.trial039.stdout:638: finishing write tid 4 to trial03942481-13 2026-02-25T08:44:26.134 INFO:tasks.rados.rados.0.trial039.stdout:638: oid 13 updating version 0 to 107 2026-02-25T08:44:26.134 INFO:tasks.rados.rados.0.trial039.stdout:638: oid 13 version 107 is already newer than 107 2026-02-25T08:44:26.134 INFO:tasks.rados.rados.0.trial039.stdout:638: oid 13 version 107 is already newer than 107 2026-02-25T08:44:26.134 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 13 v 107 (ObjNum 306 snap 82 seq_num 306) dirty exists 2026-02-25T08:44:26.134 INFO:tasks.rados.rados.0.trial039.stdout:638: left oid 13 (ObjNum 306 snap 82 seq_num 306) 2026-02-25T08:44:26.140 INFO:tasks.rados.rados.0.trial039.stdout:638: done (2 left) 2026-02-25T08:44:26.141 INFO:tasks.rados.rados.0.trial039.stdout:640: done (1 left) 2026-02-25T08:44:26.141 INFO:tasks.rados.rados.0.trial039.stdout:641: read oid 8 snap -1 2026-02-25T08:44:26.141 INFO:tasks.rados.rados.0.trial039.stdout:641: expect (ObjNum 215 snap 56 seq_num 215) 2026-02-25T08:44:26.141 INFO:tasks.rados.rados.0.trial039.stdout:642: read oid 15 snap -1 2026-02-25T08:44:26.141 INFO:tasks.rados.rados.0.trial039.stdout:642: expect deleted 2026-02-25T08:44:26.141 INFO:tasks.rados.rados.0.trial039.stdout:643: write oid 35 current snap is 82 2026-02-25T08:44:26.141 INFO:tasks.rados.rados.0.trial039.stdout:643: seq_num 309 ranges {793363=706061,2282230=796095} 2026-02-25T08:44:26.146 INFO:tasks.rados.rados.0.trial039.stdout:643: writing trial03942481-35 from 793363 to 1499424 tid 1 2026-02-25T08:44:26.152 INFO:tasks.rados.rados.0.trial039.stdout:643: writing trial03942481-35 from 2282230 to 3078325 tid 2 2026-02-25T08:44:26.152 INFO:tasks.rados.rados.0.trial039.stdout:644: write (excl) oid 47 current snap is 82 2026-02-25T08:44:26.152 INFO:tasks.rados.rados.0.trial039.stdout:644: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2026-02-25T08:44:26.157 INFO:tasks.rados.rados.0.trial039.stdout:644: writing trial03942481-47 from 410170 to 1191480 tid 1 2026-02-25T08:44:26.162 INFO:tasks.rados.rados.0.trial039.stdout:644: writing trial03942481-47 from 1940712 to 2596818 tid 2 2026-02-25T08:44:26.164 INFO:tasks.rados.rados.0.trial039.stdout:644: writing trial03942481-47 from 3163363 to 3210170 tid 3 2026-02-25T08:44:26.164 INFO:tasks.rados.rados.0.trial039.stdout:645: snap_remove snap 80 2026-02-25T08:44:26.359 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:26.371 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:26.432 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:26.435 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:26.449 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:26.447+0000 7f60483ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:26.452 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:26.451+0000 7f60483ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:26.478 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:26.494 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:27.084 INFO:tasks.rados.rados.0.trial039.stdout:645: done (5 left) 2026-02-25T08:44:27.084 INFO:tasks.rados.rados.0.trial039.stdout:646: snap_remove snap 66 2026-02-25T08:44:28.084 INFO:tasks.rados.rados.0.trial039.stdout:646: done (5 left) 2026-02-25T08:44:28.085 INFO:tasks.rados.rados.0.trial039.stdout:647: read oid 17 snap -1 2026-02-25T08:44:28.085 INFO:tasks.rados.rados.0.trial039.stdout:647: expect deleted 2026-02-25T08:44:28.085 INFO:tasks.rados.rados.0.trial039.stdout:648: write oid 31 current snap is 82 2026-02-25T08:44:28.085 INFO:tasks.rados.rados.0.trial039.stdout:648: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2026-02-25T08:44:28.087 INFO:tasks.rados.rados.0.trial039.stdout:648: writing trial03942481-31 from 426977 to 883536 tid 1 2026-02-25T08:44:28.093 INFO:tasks.rados.rados.0.trial039.stdout:648: writing trial03942481-31 from 1482841 to 2282605 tid 2 2026-02-25T08:44:28.097 INFO:tasks.rados.rados.0.trial039.stdout:648: writing trial03942481-31 from 2958080 to 3226977 tid 3 2026-02-25T08:44:28.097 INFO:tasks.rados.rados.0.trial039.stdout:649: write (excl) oid 34 current snap is 82 2026-02-25T08:44:28.097 INFO:tasks.rados.rados.0.trial039.stdout:649: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2026-02-25T08:44:28.100 INFO:tasks.rados.rados.0.trial039.stdout:649: writing trial03942481-34 from 443784 to 1091945 tid 1 2026-02-25T08:44:28.105 INFO:tasks.rados.rados.0.trial039.stdout:649: writing trial03942481-34 from 1657676 to 2317451 tid 2 2026-02-25T08:44:28.109 INFO:tasks.rados.rados.0.trial039.stdout:649: writing trial03942481-34 from 2818209 to 3243784 tid 3 2026-02-25T08:44:28.109 INFO:tasks.rados.rados.0.trial039.stdout:650: snap_remove snap 81 2026-02-25T08:44:28.495 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:28.573 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:28.570+0000 7f559ebe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:28.575 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:28.574+0000 7f559ebe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:28.602 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:28.617 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:29.091 INFO:tasks.rados.rados.0.trial039.stdout:650: done (8 left) 2026-02-25T08:44:29.091 INFO:tasks.rados.rados.0.trial039.stdout:651: write (excl) oid 39 current snap is 82 2026-02-25T08:44:29.091 INFO:tasks.rados.rados.0.trial039.stdout:651: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2026-02-25T08:44:29.095 INFO:tasks.rados.rados.0.trial039.stdout:651: writing trial03942481-39 from 460591 to 1184001 tid 1 2026-02-25T08:44:29.099 INFO:tasks.rados.rados.0.trial039.stdout:651: writing trial03942481-39 from 1716158 to 2119591 tid 2 2026-02-25T08:44:29.104 INFO:tasks.rados.rados.0.trial039.stdout:651: writing trial03942481-39 from 2729279 to 3169419 tid 3 2026-02-25T08:44:29.104 INFO:tasks.rados.rados.0.trial039.stdout:642: done (8 left) 2026-02-25T08:44:29.104 INFO:tasks.rados.rados.0.trial039.stdout:652: write (excl) oid 29 current snap is 82 2026-02-25T08:44:29.104 INFO:tasks.rados.rados.0.trial039.stdout:652: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2026-02-25T08:44:29.109 INFO:tasks.rados.rados.0.trial039.stdout:652: writing trial03942481-29 from 477398 to 1276057 tid 1 2026-02-25T08:44:29.114 INFO:tasks.rados.rados.0.trial039.stdout:652: writing trial03942481-29 from 1774640 to 2438084 tid 2 2026-02-25T08:44:29.117 INFO:tasks.rados.rados.0.trial039.stdout:652: writing trial03942481-29 from 2873055 to 3277398 tid 3 2026-02-25T08:44:29.118 INFO:tasks.rados.rados.0.trial039.stdout:653: read oid 25 snap -1 2026-02-25T08:44:29.118 INFO:tasks.rados.rados.0.trial039.stdout:653: expect (ObjNum 126 snap 18 seq_num 126) 2026-02-25T08:44:29.118 INFO:tasks.rados.rados.0.trial039.stdout:654: delete oid 4 current snap is 82 2026-02-25T08:44:29.118 INFO:tasks.rados.rados.0.trial039.stdout:639: finishing write tid 2 to trial03942481-46 2026-02-25T08:44:29.118 INFO:tasks.rados.rados.0.trial039.stdout:643: finishing write tid 1 to trial03942481-35 2026-02-25T08:44:29.118 INFO:tasks.rados.rados.0.trial039.stdout:641: expect (ObjNum 215 snap 56 seq_num 215) 2026-02-25T08:44:29.218 INFO:tasks.rados.rados.0.trial039.stdout:639: finishing write tid 3 to trial03942481-46 2026-02-25T08:44:29.218 INFO:tasks.rados.rados.0.trial039.stdout:639: finishing write tid 4 to trial03942481-46 2026-02-25T08:44:29.218 INFO:tasks.rados.rados.0.trial039.stdout:639: finishing write tid 5 to trial03942481-46 2026-02-25T08:44:29.218 INFO:tasks.rados.rados.0.trial039.stdout:639: oid 46 updating version 0 to 86 2026-02-25T08:44:29.218 INFO:tasks.rados.rados.0.trial039.stdout:639: oid 46 version 86 is already newer than 83 2026-02-25T08:44:29.218 INFO:tasks.rados.rados.0.trial039.stdout:639: oid 46 version 86 is already newer than 86 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:639: oid 46 version 86 is already newer than 86 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 46 v 86 (ObjNum 307 snap 82 seq_num 307) dirty exists 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:639: left oid 46 (ObjNum 307 snap 82 seq_num 307) 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:643: finishing write tid 2 to trial03942481-35 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:639: done (10 left) 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:641: done (9 left) 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:654: done (8 left) 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:655: read oid 12 snap -1 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:655: expect deleted 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:656: read oid 33 snap -1 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:656: expect deleted 2026-02-25T08:44:29.219 INFO:tasks.rados.rados.0.trial039.stdout:657: delete oid 37 current snap is 82 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:643: finishing write tid 3 to trial03942481-35 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:643: finishing write tid 4 to trial03942481-35 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:643: oid 35 updating version 0 to 69 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:643: oid 35 version 69 is already newer than 69 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:643: oid 35 version 69 is already newer than 69 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 35 v 69 (ObjNum 309 snap 82 seq_num 309) dirty exists 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:643: left oid 35 (ObjNum 309 snap 82 seq_num 309) 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:644: finishing write tid 1 to trial03942481-47 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:644: finishing write tid 2 to trial03942481-47 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:644: finishing write tid 3 to trial03942481-47 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:644: finishing write tid 4 to trial03942481-47 2026-02-25T08:44:29.220 INFO:tasks.rados.rados.0.trial039.stdout:644: finishing write tid 5 to trial03942481-47 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:644: oid 47 updating version 0 to 74 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:644: oid 47 version 74 is already newer than 74 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:644: oid 47 version 74 is already newer than 71 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:644: oid 47 version 74 is already newer than 74 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 47 v 74 (ObjNum 310 snap 82 seq_num 310) dirty exists 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:644: left oid 47 (ObjNum 310 snap 82 seq_num 310) 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:648: finishing write tid 1 to trial03942481-31 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:648: finishing write tid 2 to trial03942481-31 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:649: finishing write tid 1 to trial03942481-34 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:648: finishing write tid 3 to trial03942481-31 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:649: finishing write tid 2 to trial03942481-34 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:648: finishing write tid 4 to trial03942481-31 2026-02-25T08:44:29.221 INFO:tasks.rados.rados.0.trial039.stdout:648: finishing write tid 5 to trial03942481-31 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:648: oid 31 updating version 0 to 237 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:648: oid 31 version 237 is already newer than 237 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:648: oid 31 updating version 237 to 240 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:648: oid 31 version 240 is already newer than 237 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 240 (ObjNum 311 snap 82 seq_num 311) dirty exists 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:648: left oid 31 (ObjNum 311 snap 82 seq_num 311) 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:649: finishing write tid 3 to trial03942481-34 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:649: finishing write tid 4 to trial03942481-34 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:649: finishing write tid 5 to trial03942481-34 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:649: oid 34 updating version 0 to 141 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:649: oid 34 version 141 is already newer than 141 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:649: oid 34 version 141 is already newer than 138 2026-02-25T08:44:29.222 INFO:tasks.rados.rados.0.trial039.stdout:649: oid 34 version 141 is already newer than 141 2026-02-25T08:44:29.223 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 141 (ObjNum 312 snap 82 seq_num 312) dirty exists 2026-02-25T08:44:29.223 INFO:tasks.rados.rados.0.trial039.stdout:649: left oid 34 (ObjNum 312 snap 82 seq_num 312) 2026-02-25T08:44:29.223 INFO:tasks.rados.rados.0.trial039.stdout:651: finishing write tid 1 to trial03942481-39 2026-02-25T08:44:29.223 INFO:tasks.rados.rados.0.trial039.stdout:651: finishing write tid 2 to trial03942481-39 2026-02-25T08:44:29.223 INFO:tasks.rados.rados.0.trial039.stdout:653: expect (ObjNum 126 snap 18 seq_num 126) 2026-02-25T08:44:29.283 INFO:tasks.rados.rados.0.trial039.stdout:651: finishing write tid 3 to trial03942481-39 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:643: done (10 left) 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:644: done (9 left) 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:647: done (8 left) 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:648: done (7 left) 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:649: done (6 left) 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:653: done (5 left) 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:657: done (4 left) 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:658: snap_create 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:652: finishing write tid 1 to trial03942481-29 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:651: finishing write tid 4 to trial03942481-39 2026-02-25T08:44:29.284 INFO:tasks.rados.rados.0.trial039.stdout:651: finishing write tid 5 to trial03942481-39 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:651: oid 39 updating version 0 to 241 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:651: oid 39 updating version 241 to 244 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:651: oid 39 version 244 is already newer than 244 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:651: oid 39 version 244 is already newer than 244 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 39 v 244 (ObjNum 313 snap 82 seq_num 313) dirty exists 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:651: left oid 39 (ObjNum 313 snap 82 seq_num 313) 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:652: finishing write tid 2 to trial03942481-29 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:652: finishing write tid 3 to trial03942481-29 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:652: finishing write tid 4 to trial03942481-29 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:652: finishing write tid 5 to trial03942481-29 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:652: oid 29 updating version 0 to 58 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:652: oid 29 updating version 58 to 61 2026-02-25T08:44:29.285 INFO:tasks.rados.rados.0.trial039.stdout:652: oid 29 version 61 is already newer than 61 2026-02-25T08:44:29.286 INFO:tasks.rados.rados.0.trial039.stdout:652: oid 29 version 61 is already newer than 61 2026-02-25T08:44:29.286 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 29 v 61 (ObjNum 314 snap 82 seq_num 314) dirty exists 2026-02-25T08:44:29.286 INFO:tasks.rados.rados.0.trial039.stdout:652: left oid 29 (ObjNum 314 snap 82 seq_num 314) 2026-02-25T08:44:30.094 INFO:tasks.rados.rados.0.trial039.stdout:651: done (4 left) 2026-02-25T08:44:30.095 INFO:tasks.rados.rados.0.trial039.stdout:652: done (3 left) 2026-02-25T08:44:30.095 INFO:tasks.rados.rados.0.trial039.stdout:655: done (2 left) 2026-02-25T08:44:30.095 INFO:tasks.rados.rados.0.trial039.stdout:656: done (1 left) 2026-02-25T08:44:30.095 INFO:tasks.rados.rados.0.trial039.stdout:658: done (0 left) 2026-02-25T08:44:30.095 INFO:tasks.rados.rados.0.trial039.stdout:659: delete oid 5 current snap is 83 2026-02-25T08:44:30.139 INFO:tasks.rados.rados.0.trial039.stdout:659: done (0 left) 2026-02-25T08:44:30.139 INFO:tasks.rados.rados.0.trial039.stdout:660: write oid 2 current snap is 83 2026-02-25T08:44:30.139 INFO:tasks.rados.rados.0.trial039.stdout:660: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2026-02-25T08:44:30.144 INFO:tasks.rados.rados.0.trial039.stdout:660: writing trial03942481-2 from 544626 to 1244281 tid 1 2026-02-25T08:44:30.149 INFO:tasks.rados.rados.0.trial039.stdout:660: writing trial03942481-2 from 1892215 to 2446644 tid 2 2026-02-25T08:44:30.152 INFO:tasks.rados.rados.0.trial039.stdout:660: writing trial03942481-2 from 3150041 to 3344626 tid 3 2026-02-25T08:44:30.152 INFO:tasks.rados.rados.0.trial039.stdout:661: write (excl) oid 28 current snap is 83 2026-02-25T08:44:30.152 INFO:tasks.rados.rados.0.trial039.stdout:661: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2026-02-25T08:44:30.157 INFO:tasks.rados.rados.0.trial039.stdout:661: writing trial03942481-28 from 561433 to 1336337 tid 1 2026-02-25T08:44:30.161 INFO:tasks.rados.rados.0.trial039.stdout:661: writing trial03942481-28 from 1950697 to 2365137 tid 2 2026-02-25T08:44:30.165 INFO:tasks.rados.rados.0.trial039.stdout:661: writing trial03942481-28 from 2777464 to 3361433 tid 3 2026-02-25T08:44:30.165 INFO:tasks.rados.rados.0.trial039.stdout:662: write (excl) oid 38 current snap is 83 2026-02-25T08:44:30.165 INFO:tasks.rados.rados.0.trial039.stdout:662: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2026-02-25T08:44:30.169 INFO:tasks.rados.rados.0.trial039.stdout:662: writing trial03942481-38 from 578240 to 1144746 tid 1 2026-02-25T08:44:30.174 INFO:tasks.rados.rados.0.trial039.stdout:662: writing trial03942481-38 from 1609179 to 2167277 tid 2 2026-02-25T08:44:30.179 INFO:tasks.rados.rados.0.trial039.stdout:662: writing trial03942481-38 from 2804887 to 3356637 tid 3 2026-02-25T08:44:30.179 INFO:tasks.rados.rados.0.trial039.stdout:663: delete oid 3 current snap is 83 2026-02-25T08:44:30.179 INFO:tasks.rados.rados.0.trial039.stdout:660: finishing write tid 1 to trial03942481-2 2026-02-25T08:44:30.179 INFO:tasks.rados.rados.0.trial039.stdout:661: finishing write tid 1 to trial03942481-28 2026-02-25T08:44:30.194 INFO:tasks.rados.rados.0.trial039.stdout:661: finishing write tid 2 to trial03942481-28 2026-02-25T08:44:30.201 INFO:tasks.rados.rados.0.trial039.stdout:661: finishing write tid 3 to trial03942481-28 2026-02-25T08:44:30.203 INFO:tasks.rados.rados.0.trial039.stdout:663: done (3 left) 2026-02-25T08:44:30.203 INFO:tasks.rados.rados.0.trial039.stdout:664: delete oid 31 current snap is 83 2026-02-25T08:44:30.206 INFO:tasks.rados.rados.0.trial039.stdout:662: finishing write tid 1 to trial03942481-38 2026-02-25T08:44:30.206 INFO:tasks.rados.rados.0.trial039.stdout:661: finishing write tid 4 to trial03942481-28 2026-02-25T08:44:30.208 INFO:tasks.rados.rados.0.trial039.stdout:660: finishing write tid 2 to trial03942481-2 2026-02-25T08:44:30.211 INFO:tasks.rados.rados.0.trial039.stdout:661: finishing write tid 5 to trial03942481-28 2026-02-25T08:44:30.211 INFO:tasks.rados.rados.0.trial039.stdout:661: oid 28 updating version 0 to 90 2026-02-25T08:44:30.211 INFO:tasks.rados.rados.0.trial039.stdout:661: oid 28 version 90 is already newer than 87 2026-02-25T08:44:30.211 INFO:tasks.rados.rados.0.trial039.stdout:661: oid 28 version 90 is already newer than 90 2026-02-25T08:44:30.211 INFO:tasks.rados.rados.0.trial039.stdout:661: oid 28 version 90 is already newer than 90 2026-02-25T08:44:30.212 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 28 v 90 (ObjNum 319 snap 83 seq_num 319) dirty exists 2026-02-25T08:44:30.212 INFO:tasks.rados.rados.0.trial039.stdout:661: left oid 28 (ObjNum 319 snap 83 seq_num 319) 2026-02-25T08:44:30.230 INFO:tasks.rados.rados.0.trial039.stdout:660: finishing write tid 3 to trial03942481-2 2026-02-25T08:44:30.234 INFO:tasks.rados.rados.0.trial039.stdout:662: finishing write tid 2 to trial03942481-38 2026-02-25T08:44:30.243 INFO:tasks.rados.rados.0.trial039.stdout:660: finishing write tid 4 to trial03942481-2 2026-02-25T08:44:30.245 INFO:tasks.rados.rados.0.trial039.stdout:662: finishing write tid 3 to trial03942481-38 2026-02-25T08:44:30.248 INFO:tasks.rados.rados.0.trial039.stdout:660: finishing write tid 5 to trial03942481-2 2026-02-25T08:44:30.248 INFO:tasks.rados.rados.0.trial039.stdout:660: oid 2 updating version 0 to 116 2026-02-25T08:44:30.248 INFO:tasks.rados.rados.0.trial039.stdout:660: oid 2 version 116 is already newer than 116 2026-02-25T08:44:30.249 INFO:tasks.rados.rados.0.trial039.stdout:660: oid 2 version 116 is already newer than 116 2026-02-25T08:44:30.249 INFO:tasks.rados.rados.0.trial039.stdout:660: oid 2 version 116 is already newer than 116 2026-02-25T08:44:30.249 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 116 (ObjNum 318 snap 83 seq_num 318) dirty exists 2026-02-25T08:44:30.249 INFO:tasks.rados.rados.0.trial039.stdout:660: left oid 2 (ObjNum 318 snap 83 seq_num 318) 2026-02-25T08:44:30.262 INFO:tasks.rados.rados.0.trial039.stdout:660: done (3 left) 2026-02-25T08:44:30.262 INFO:tasks.rados.rados.0.trial039.stdout:661: done (2 left) 2026-02-25T08:44:30.262 INFO:tasks.rados.rados.0.trial039.stdout:664: done (1 left) 2026-02-25T08:44:30.263 INFO:tasks.rados.rados.0.trial039.stdout:665: snap_create 2026-02-25T08:44:30.263 INFO:tasks.rados.rados.0.trial039.stdout:662: finishing write tid 4 to trial03942481-38 2026-02-25T08:44:30.270 INFO:tasks.rados.rados.0.trial039.stdout:662: finishing write tid 5 to trial03942481-38 2026-02-25T08:44:30.270 INFO:tasks.rados.rados.0.trial039.stdout:662: oid 38 updating version 0 to 110 2026-02-25T08:44:30.271 INFO:tasks.rados.rados.0.trial039.stdout:662: oid 38 updating version 110 to 113 2026-02-25T08:44:30.271 INFO:tasks.rados.rados.0.trial039.stdout:662: oid 38 version 113 is already newer than 113 2026-02-25T08:44:30.271 INFO:tasks.rados.rados.0.trial039.stdout:662: oid 38 version 113 is already newer than 113 2026-02-25T08:44:30.271 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 38 v 113 (ObjNum 320 snap 83 seq_num 320) dirty exists 2026-02-25T08:44:30.271 INFO:tasks.rados.rados.0.trial039.stdout:662: left oid 38 (ObjNum 320 snap 83 seq_num 320) 2026-02-25T08:44:30.618 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:30.695 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:30.693+0000 7f6d26a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:30.698 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:30.697+0000 7f6d26a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:30.724 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:30.741 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:31.098 INFO:tasks.rados.rados.0.trial039.stdout:662: done (1 left) 2026-02-25T08:44:31.098 INFO:tasks.rados.rados.0.trial039.stdout:665: done (0 left) 2026-02-25T08:44:31.098 INFO:tasks.rados.rados.0.trial039.stdout:666: read oid 42 snap -1 2026-02-25T08:44:31.098 INFO:tasks.rados.rados.0.trial039.stdout:666: expect (ObjNum 256 snap 71 seq_num 256) 2026-02-25T08:44:31.098 INFO:tasks.rados.rados.0.trial039.stdout:667: write oid 28 current snap is 84 2026-02-25T08:44:31.099 INFO:tasks.rados.rados.0.trial039.stdout:667: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2026-02-25T08:44:31.103 INFO:tasks.rados.rados.0.trial039.stdout:667: writing trial03942481-28 from 628661 to 1420914 tid 1 2026-02-25T08:44:31.108 INFO:tasks.rados.rados.0.trial039.stdout:667: writing trial03942481-28 from 2184625 to 2606403 tid 2 2026-02-25T08:44:31.111 INFO:tasks.rados.rados.0.trial039.stdout:667: writing trial03942481-28 from 3403509 to 3428661 tid 3 2026-02-25T08:44:31.112 INFO:tasks.rados.rados.0.trial039.stdout:668: snap_remove snap 78 2026-02-25T08:44:31.437 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:31.458 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~176s 2026-02-25T08:44:31.510 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:31.513 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:32.101 INFO:tasks.rados.rados.0.trial039.stdout:668: done (2 left) 2026-02-25T08:44:32.101 INFO:tasks.rados.rados.0.trial039.stdout:669: write (excl) oid 29 current snap is 84 2026-02-25T08:44:32.102 INFO:tasks.rados.rados.0.trial039.stdout:669: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2026-02-25T08:44:32.105 INFO:tasks.rados.rados.0.trial039.stdout:669: writing trial03942481-29 from 645468 to 1112970 tid 1 2026-02-25T08:44:32.110 INFO:tasks.rados.rados.0.trial039.stdout:669: writing trial03942481-29 from 1726754 to 2292190 tid 2 2026-02-25T08:44:32.115 INFO:tasks.rados.rados.0.trial039.stdout:669: writing trial03942481-29 from 2798226 to 3395064 tid 3 2026-02-25T08:44:32.115 INFO:tasks.rados.rados.0.trial039.stdout:670: snap_create 2026-02-25T08:44:32.115 INFO:tasks.rados.rados.0.trial039.stdout:666: expect (ObjNum 256 snap 71 seq_num 256) 2026-02-25T08:44:32.146 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:32.166 INFO:tasks.rados.rados.0.trial039.stdout:667: finishing write tid 1 to trial03942481-28 2026-02-25T08:44:32.167 INFO:tasks.rados.rados.0.trial039.stdout:667: finishing write tid 2 to trial03942481-28 2026-02-25T08:44:32.167 INFO:tasks.rados.rados.0.trial039.stdout:667: finishing write tid 3 to trial03942481-28 2026-02-25T08:44:32.167 INFO:tasks.rados.rados.0.trial039.stdout:667: finishing write tid 4 to trial03942481-28 2026-02-25T08:44:32.167 INFO:tasks.rados.rados.0.trial039.stdout:667: finishing write tid 5 to trial03942481-28 2026-02-25T08:44:32.167 INFO:tasks.rados.rados.0.trial039.stdout:667: oid 28 updating version 0 to 95 2026-02-25T08:44:32.167 INFO:tasks.rados.rados.0.trial039.stdout:667: oid 28 version 95 is already newer than 95 2026-02-25T08:44:32.167 INFO:tasks.rados.rados.0.trial039.stdout:667: oid 28 version 95 is already newer than 95 2026-02-25T08:44:32.167 INFO:tasks.rados.rados.0.trial039.stdout:667: oid 28 version 95 is already newer than 95 2026-02-25T08:44:32.167 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 28 v 95 (ObjNum 323 snap 84 seq_num 323) dirty exists 2026-02-25T08:44:32.167 INFO:tasks.rados.rados.0.trial039.stdout:667: left oid 28 (ObjNum 323 snap 84 seq_num 323) 2026-02-25T08:44:32.168 INFO:tasks.rados.rados.0.trial039.stdout:669: finishing write tid 1 to trial03942481-29 2026-02-25T08:44:32.168 INFO:tasks.rados.rados.0.trial039.stdout:669: finishing write tid 2 to trial03942481-29 2026-02-25T08:44:32.169 INFO:tasks.rados.rados.0.trial039.stdout:669: finishing write tid 3 to trial03942481-29 2026-02-25T08:44:32.175 INFO:tasks.rados.rados.0.trial039.stdout:669: finishing write tid 4 to trial03942481-29 2026-02-25T08:44:32.177 INFO:tasks.rados.rados.0.trial039.stdout:669: finishing write tid 5 to trial03942481-29 2026-02-25T08:44:32.177 INFO:tasks.rados.rados.0.trial039.stdout:669: oid 29 updating version 0 to 66 2026-02-25T08:44:32.177 INFO:tasks.rados.rados.0.trial039.stdout:669: oid 29 version 66 is already newer than 66 2026-02-25T08:44:32.177 INFO:tasks.rados.rados.0.trial039.stdout:669: oid 29 version 66 is already newer than 66 2026-02-25T08:44:32.177 INFO:tasks.rados.rados.0.trial039.stdout:669: oid 29 version 66 is already newer than 63 2026-02-25T08:44:32.177 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 29 v 66 (ObjNum 324 snap 84 seq_num 324) dirty exists 2026-02-25T08:44:32.177 INFO:tasks.rados.rados.0.trial039.stdout:669: left oid 29 (ObjNum 324 snap 84 seq_num 324) 2026-02-25T08:44:32.741 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:32.818 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:32.815+0000 7f7d2a6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:32.820 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:32.819+0000 7f7d2a6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:32.847 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:32.864 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:33.075 INFO:tasks.rados.rados.0.trial039.stdout:666: done (3 left) 2026-02-25T08:44:33.076 INFO:tasks.rados.rados.0.trial039.stdout:667: done (2 left) 2026-02-25T08:44:33.076 INFO:tasks.rados.rados.0.trial039.stdout:669: done (1 left) 2026-02-25T08:44:33.076 INFO:tasks.rados.rados.0.trial039.stdout:670: done (0 left) 2026-02-25T08:44:33.076 INFO:tasks.rados.rados.0.trial039.stdout:671: snap_remove snap 82 2026-02-25T08:44:34.076 INFO:tasks.rados.rados.0.trial039.stdout:671: done (0 left) 2026-02-25T08:44:34.076 INFO:tasks.rados.rados.0.trial039.stdout:672: read oid 12 snap -1 2026-02-25T08:44:34.076 INFO:tasks.rados.rados.0.trial039.stdout:672: expect deleted 2026-02-25T08:44:34.076 INFO:tasks.rados.rados.0.trial039.stdout:673: write (excl) oid 27 current snap is 85 2026-02-25T08:44:34.076 INFO:tasks.rados.rados.0.trial039.stdout:673: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2026-02-25T08:44:34.080 INFO:tasks.rados.rados.0.trial039.stdout:673: writing trial03942481-27 from 662275 to 1205026 tid 1 2026-02-25T08:44:34.084 INFO:tasks.rados.rados.0.trial039.stdout:673: writing trial03942481-27 from 1785236 to 2210683 tid 2 2026-02-25T08:44:34.090 INFO:tasks.rados.rados.0.trial039.stdout:673: writing trial03942481-27 from 2942002 to 3462275 tid 3 2026-02-25T08:44:34.090 INFO:tasks.rados.rados.0.trial039.stdout:674: read oid 20 snap -1 2026-02-25T08:44:34.090 INFO:tasks.rados.rados.0.trial039.stdout:674: expect (ObjNum 305 snap 81 seq_num 305) 2026-02-25T08:44:34.090 INFO:tasks.rados.rados.0.trial039.stdout:675: write (excl) oid 18 current snap is 85 2026-02-25T08:44:34.090 INFO:tasks.rados.rados.0.trial039.stdout:675: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2026-02-25T08:44:34.095 INFO:tasks.rados.rados.0.trial039.stdout:675: writing trial03942481-18 from 679082 to 1297082 tid 1 2026-02-25T08:44:34.099 INFO:tasks.rados.rados.0.trial039.stdout:675: writing trial03942481-18 from 1843718 to 2412823 tid 2 2026-02-25T08:44:34.104 INFO:tasks.rados.rados.0.trial039.stdout:675: writing trial03942481-18 from 2853072 to 3479082 tid 3 2026-02-25T08:44:34.104 INFO:tasks.rados.rados.0.trial039.stdout:676: write oid 33 current snap is 85 2026-02-25T08:44:34.104 INFO:tasks.rados.rados.0.trial039.stdout:676: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2026-02-25T08:44:34.108 INFO:tasks.rados.rados.0.trial039.stdout:676: writing trial03942481-33 from 695889 to 1105491 tid 1 2026-02-25T08:44:34.112 INFO:tasks.rados.rados.0.trial039.stdout:676: writing trial03942481-33 from 1618553 to 2331316 tid 2 2026-02-25T08:44:34.117 INFO:tasks.rados.rados.0.trial039.stdout:676: writing trial03942481-33 from 2996848 to 3495889 tid 3 2026-02-25T08:44:34.118 INFO:tasks.rados.rados.0.trial039.stdout:677: snap_remove snap 67 2026-02-25T08:44:34.864 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:34.942 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:34.939+0000 7fe52fd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:34.944 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:34.943+0000 7fe52ea82640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:34.971 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:34.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:35.107 INFO:tasks.rados.rados.0.trial039.stdout:677: done (5 left) 2026-02-25T08:44:35.107 INFO:tasks.rados.rados.0.trial039.stdout:678: snap_create 2026-02-25T08:44:35.107 INFO:tasks.rados.rados.0.trial039.stdout:675: finishing write tid 1 to trial03942481-18 2026-02-25T08:44:35.107 INFO:tasks.rados.rados.0.trial039.stdout:676: finishing write tid 1 to trial03942481-33 2026-02-25T08:44:35.107 INFO:tasks.rados.rados.0.trial039.stdout:676: finishing write tid 2 to trial03942481-33 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:673: finishing write tid 1 to trial03942481-27 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:675: finishing write tid 2 to trial03942481-18 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:676: finishing write tid 3 to trial03942481-33 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:675: finishing write tid 3 to trial03942481-18 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:675: finishing write tid 4 to trial03942481-18 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:675: finishing write tid 5 to trial03942481-18 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:675: oid 18 updating version 0 to 114 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:675: oid 18 updating version 114 to 117 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:675: oid 18 version 117 is already newer than 117 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:675: oid 18 version 117 is already newer than 117 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 18 v 117 (ObjNum 326 snap 85 seq_num 326) dirty exists 2026-02-25T08:44:35.108 INFO:tasks.rados.rados.0.trial039.stdout:675: left oid 18 (ObjNum 326 snap 85 seq_num 326) 2026-02-25T08:44:35.109 INFO:tasks.rados.rados.0.trial039.stdout:673: finishing write tid 2 to trial03942481-27 2026-02-25T08:44:35.109 INFO:tasks.rados.rados.0.trial039.stdout:676: finishing write tid 4 to trial03942481-33 2026-02-25T08:44:35.109 INFO:tasks.rados.rados.0.trial039.stdout:674: expect (ObjNum 305 snap 81 seq_num 305) 2026-02-25T08:44:35.155 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 3042606 2026-02-25T08:44:35.219 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:35.282 INFO:tasks.rados.rados.0.trial039.stdout:676: finishing write tid 5 to trial03942481-33 2026-02-25T08:44:35.282 INFO:tasks.rados.rados.0.trial039.stdout:676: oid 33 updating version 0 to 99 2026-02-25T08:44:35.283 INFO:tasks.rados.rados.0.trial039.stdout:676: oid 33 version 99 is already newer than 99 2026-02-25T08:44:35.283 INFO:tasks.rados.rados.0.trial039.stdout:676: oid 33 version 99 is already newer than 99 2026-02-25T08:44:35.283 INFO:tasks.rados.rados.0.trial039.stdout:676: oid 33 version 99 is already newer than 98 2026-02-25T08:44:35.283 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 33 v 99 (ObjNum 327 snap 85 seq_num 327) dirty exists 2026-02-25T08:44:35.283 INFO:tasks.rados.rados.0.trial039.stdout:676: left oid 33 (ObjNum 327 snap 85 seq_num 327) 2026-02-25T08:44:35.283 INFO:tasks.rados.rados.0.trial039.stdout:673: finishing write tid 3 to trial03942481-27 2026-02-25T08:44:35.283 INFO:tasks.rados.rados.0.trial039.stdout:673: finishing write tid 4 to trial03942481-27 2026-02-25T08:44:35.283 INFO:tasks.rados.rados.0.trial039.stdout:673: finishing write tid 5 to trial03942481-27 2026-02-25T08:44:35.283 INFO:tasks.rados.rados.0.trial039.stdout:673: oid 27 updating version 0 to 86 2026-02-25T08:44:35.283 INFO:tasks.rados.rados.0.trial039.stdout:673: oid 27 version 86 is already newer than 86 2026-02-25T08:44:35.284 INFO:tasks.rados.rados.0.trial039.stdout:673: oid 27 version 86 is already newer than 83 2026-02-25T08:44:35.284 INFO:tasks.rados.rados.0.trial039.stdout:673: oid 27 version 86 is already newer than 86 2026-02-25T08:44:35.284 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 27 v 86 (ObjNum 325 snap 85 seq_num 325) dirty exists 2026-02-25T08:44:35.284 INFO:tasks.rados.rados.0.trial039.stdout:673: left oid 27 (ObjNum 325 snap 85 seq_num 325) 2026-02-25T08:44:36.109 INFO:tasks.rados.rados.0.trial039.stdout:672: done (5 left) 2026-02-25T08:44:36.110 INFO:tasks.rados.rados.0.trial039.stdout:673: done (4 left) 2026-02-25T08:44:36.110 INFO:tasks.rados.rados.0.trial039.stdout:674: done (3 left) 2026-02-25T08:44:36.110 INFO:tasks.rados.rados.0.trial039.stdout:675: done (2 left) 2026-02-25T08:44:36.110 INFO:tasks.rados.rados.0.trial039.stdout:676: done (1 left) 2026-02-25T08:44:36.110 INFO:tasks.rados.rados.0.trial039.stdout:678: done (0 left) 2026-02-25T08:44:36.110 INFO:tasks.rados.rados.0.trial039.stdout:679: snap_remove snap 79 2026-02-25T08:44:36.514 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:36.587 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:36.590 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:36.590 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:36.962 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~182s 2026-02-25T08:44:36.990 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:37.067 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:37.064+0000 7f2ec501a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:37.069 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:37.068+0000 7f2ec4819640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:37.096 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:37.114 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:37.114 INFO:tasks.rados.rados.0.trial039.stdout:679: done (0 left) 2026-02-25T08:44:37.114 INFO:tasks.rados.rados.0.trial039.stdout:680: delete oid 19 current snap is 86 2026-02-25T08:44:37.152 INFO:tasks.rados.rados.0.trial039.stdout:680: done (0 left) 2026-02-25T08:44:37.152 INFO:tasks.rados.rados.0.trial039.stdout:681: write oid 48 current snap is 86 2026-02-25T08:44:37.152 INFO:tasks.rados.rados.0.trial039.stdout:681: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2026-02-25T08:44:37.156 INFO:tasks.rados.rados.0.trial039.stdout:681: writing trial03942481-48 from 729503 to 1289603 tid 1 2026-02-25T08:44:37.162 INFO:tasks.rados.rados.0.trial039.stdout:681: writing trial03942481-48 from 2019164 to 2735596 tid 2 2026-02-25T08:44:37.165 INFO:tasks.rados.rados.0.trial039.stdout:681: writing trial03942481-48 from 3335341 to 3529503 tid 3 2026-02-25T08:44:37.166 INFO:tasks.rados.rados.0.trial039.stdout:682: delete oid 50 current snap is 86 2026-02-25T08:44:37.168 INFO:tasks.rados.rados.0.trial039.stdout:682: done (1 left) 2026-02-25T08:44:37.168 INFO:tasks.rados.rados.0.trial039.stdout:683: snap_remove snap 83 2026-02-25T08:44:38.119 INFO:tasks.rados.rados.0.trial039.stdout:683: done (1 left) 2026-02-25T08:44:38.119 INFO:tasks.rados.rados.0.trial039.stdout:684: write oid 47 current snap is 86 2026-02-25T08:44:38.119 INFO:tasks.rados.rados.0.trial039.stdout:681: finishing write tid 1 to trial03942481-48 2026-02-25T08:44:38.119 INFO:tasks.rados.rados.0.trial039.stdout:684: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2026-02-25T08:44:38.124 INFO:tasks.rados.rados.0.trial039.stdout:684: writing trial03942481-47 from 763117 to 1473715 tid 1 2026-02-25T08:44:38.130 INFO:tasks.rados.rados.0.trial039.stdout:684: writing trial03942481-47 from 2136128 to 2856229 tid 2 2026-02-25T08:44:38.133 INFO:tasks.rados.rados.0.trial039.stdout:684: writing trial03942481-47 from 3390187 to 3563117 tid 3 2026-02-25T08:44:38.134 INFO:tasks.rados.rados.0.trial039.stdout:685: snap_remove snap 84 2026-02-25T08:44:39.115 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:39.123 INFO:tasks.rados.rados.0.trial039.stdout:685: done (2 left) 2026-02-25T08:44:39.123 INFO:tasks.rados.rados.0.trial039.stdout:686: snap_create 2026-02-25T08:44:39.123 INFO:tasks.rados.rados.0.trial039.stdout:681: finishing write tid 2 to trial03942481-48 2026-02-25T08:44:39.123 INFO:tasks.rados.rados.0.trial039.stdout:681: finishing write tid 3 to trial03942481-48 2026-02-25T08:44:39.123 INFO:tasks.rados.rados.0.trial039.stdout:681: finishing write tid 4 to trial03942481-48 2026-02-25T08:44:39.123 INFO:tasks.rados.rados.0.trial039.stdout:681: finishing write tid 5 to trial03942481-48 2026-02-25T08:44:39.123 INFO:tasks.rados.rados.0.trial039.stdout:681: oid 48 updating version 0 to 38 2026-02-25T08:44:39.123 INFO:tasks.rados.rados.0.trial039.stdout:681: oid 48 version 38 is already newer than 38 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:681: oid 48 version 38 is already newer than 38 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:681: oid 48 version 38 is already newer than 38 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 48 v 38 (ObjNum 329 snap 86 seq_num 329) dirty exists 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:681: left oid 48 (ObjNum 329 snap 86 seq_num 329) 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:684: finishing write tid 1 to trial03942481-47 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:684: finishing write tid 2 to trial03942481-47 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:684: finishing write tid 3 to trial03942481-47 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:684: finishing write tid 4 to trial03942481-47 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:684: finishing write tid 5 to trial03942481-47 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:684: oid 47 updating version 0 to 95 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:684: oid 47 version 95 is already newer than 95 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:684: oid 47 version 95 is already newer than 95 2026-02-25T08:44:39.124 INFO:tasks.rados.rados.0.trial039.stdout:684: oid 47 version 95 is already newer than 95 2026-02-25T08:44:39.125 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 47 v 95 (ObjNum 331 snap 86 seq_num 331) dirty exists 2026-02-25T08:44:39.125 INFO:tasks.rados.rados.0.trial039.stdout:684: left oid 47 (ObjNum 331 snap 86 seq_num 331) 2026-02-25T08:44:39.193 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:39.190+0000 7f45c2e51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:39.196 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:39.194+0000 7f45c2e51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:39.221 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:39.238 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:40.127 INFO:tasks.rados.rados.0.trial039.stdout:681: done (2 left) 2026-02-25T08:44:40.127 INFO:tasks.rados.rados.0.trial039.stdout:684: done (1 left) 2026-02-25T08:44:40.127 INFO:tasks.rados.rados.0.trial039.stdout:686: done (0 left) 2026-02-25T08:44:40.127 INFO:tasks.rados.rados.0.trial039.stdout:687: read oid 44 snap 73 2026-02-25T08:44:40.127 INFO:tasks.rados.rados.0.trial039.stdout:687: expect (ObjNum 252 snap 70 seq_num 252) 2026-02-25T08:44:40.127 INFO:tasks.rados.rados.0.trial039.stdout:688: write oid 47 current snap is 87 2026-02-25T08:44:40.128 INFO:tasks.rados.rados.0.trial039.stdout:688: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2026-02-25T08:44:40.132 INFO:tasks.rados.rados.0.trial039.stdout:688: writing trial03942481-47 from 779924 to 1565771 tid 1 2026-02-25T08:44:40.137 INFO:tasks.rados.rados.0.trial039.stdout:688: writing trial03942481-47 from 2078257 to 2658369 tid 2 2026-02-25T08:44:40.141 INFO:tasks.rados.rados.0.trial039.stdout:688: writing trial03942481-47 from 3301257 to 3579924 tid 3 2026-02-25T08:44:40.141 INFO:tasks.rados.rados.0.trial039.stdout:689: write (excl) oid 8 current snap is 87 2026-02-25T08:44:40.141 INFO:tasks.rados.rados.0.trial039.stdout:689: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2026-02-25T08:44:40.146 INFO:tasks.rados.rados.0.trial039.stdout:689: writing trial03942481-8 from 796731 to 1257827 tid 1 2026-02-25T08:44:40.150 INFO:tasks.rados.rados.0.trial039.stdout:689: writing trial03942481-8 from 1736739 to 2460509 tid 2 2026-02-25T08:44:40.155 INFO:tasks.rados.rados.0.trial039.stdout:689: writing trial03942481-8 from 2928680 to 3596731 tid 3 2026-02-25T08:44:40.155 INFO:tasks.rados.rados.0.trial039.stdout:690: delete oid 24 current snap is 87 2026-02-25T08:44:40.157 INFO:tasks.rados.rados.0.trial039.stdout:687: expect (ObjNum 252 snap 70 seq_num 252) 2026-02-25T08:44:40.160 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 623016 2026-02-25T08:44:40.202 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 908026 2026-02-25T08:44:40.257 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1662339 2026-02-25T08:44:40.269 INFO:tasks.rados.rados.0.trial039.stdout:689: finishing write tid 1 to trial03942481-8 2026-02-25T08:44:40.270 INFO:tasks.rados.rados.0.trial039.stdout:688: finishing write tid 1 to trial03942481-47 2026-02-25T08:44:40.270 INFO:tasks.rados.rados.0.trial039.stdout:687: done (3 left) 2026-02-25T08:44:40.270 INFO:tasks.rados.rados.0.trial039.stdout:690: done (2 left) 2026-02-25T08:44:40.270 INFO:tasks.rados.rados.0.trial039.stdout:691: snap_create 2026-02-25T08:44:40.270 INFO:tasks.rados.rados.0.trial039.stdout:689: finishing write tid 2 to trial03942481-8 2026-02-25T08:44:40.270 INFO:tasks.rados.rados.0.trial039.stdout:688: finishing write tid 2 to trial03942481-47 2026-02-25T08:44:40.270 INFO:tasks.rados.rados.0.trial039.stdout:688: finishing write tid 3 to trial03942481-47 2026-02-25T08:44:40.270 INFO:tasks.rados.rados.0.trial039.stdout:688: finishing write tid 4 to trial03942481-47 2026-02-25T08:44:40.270 INFO:tasks.rados.rados.0.trial039.stdout:688: finishing write tid 5 to trial03942481-47 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:688: oid 47 updating version 0 to 102 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:688: oid 47 version 102 is already newer than 102 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:688: oid 47 version 102 is already newer than 102 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:688: oid 47 version 102 is already newer than 102 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 47 v 102 (ObjNum 332 snap 87 seq_num 332) dirty exists 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:688: left oid 47 (ObjNum 332 snap 87 seq_num 332) 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:689: finishing write tid 3 to trial03942481-8 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:689: finishing write tid 4 to trial03942481-8 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:689: finishing write tid 5 to trial03942481-8 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:689: oid 8 updating version 0 to 104 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:689: oid 8 version 104 is already newer than 104 2026-02-25T08:44:40.271 INFO:tasks.rados.rados.0.trial039.stdout:689: oid 8 version 104 is already newer than 101 2026-02-25T08:44:40.272 INFO:tasks.rados.rados.0.trial039.stdout:689: oid 8 version 104 is already newer than 104 2026-02-25T08:44:40.272 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 8 v 104 (ObjNum 333 snap 87 seq_num 333) dirty exists 2026-02-25T08:44:40.272 INFO:tasks.rados.rados.0.trial039.stdout:689: left oid 8 (ObjNum 333 snap 87 seq_num 333) 2026-02-25T08:44:41.131 INFO:tasks.rados.rados.0.trial039.stdout:688: done (2 left) 2026-02-25T08:44:41.131 INFO:tasks.rados.rados.0.trial039.stdout:689: done (1 left) 2026-02-25T08:44:41.131 INFO:tasks.rados.rados.0.trial039.stdout:691: done (0 left) 2026-02-25T08:44:41.131 INFO:tasks.rados.rados.0.trial039.stdout:692: snap_create 2026-02-25T08:44:41.239 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:41.318 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:41.314+0000 7f4a2015c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:41.320 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:41.318+0000 7f4a2015c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:41.346 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:41.363 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:41.591 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:41.663 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:41.666 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:42.133 INFO:tasks.rados.rados.0.trial039.stdout:692: done (0 left) 2026-02-25T08:44:42.133 INFO:tasks.rados.rados.0.trial039.stdout:693: write (excl) oid 44 current snap is 89 2026-02-25T08:44:42.134 INFO:tasks.rados.rados.0.trial039.stdout:693: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2026-02-25T08:44:42.137 INFO:tasks.rados.rados.0.trial039.stdout:693: writing trial03942481-44 from 430345 to 1158292 tid 1 2026-02-25T08:44:42.142 INFO:tasks.rados.rados.0.trial039.stdout:693: writing trial03942481-44 from 1570056 to 2297495 tid 2 2026-02-25T08:44:42.146 INFO:tasks.rados.rados.0.trial039.stdout:693: writing trial03942481-44 from 2699879 to 3169768 tid 3 2026-02-25T08:44:42.146 INFO:tasks.rados.rados.0.trial039.stdout:694: delete oid 38 current snap is 89 2026-02-25T08:44:42.176 INFO:tasks.rados.rados.0.trial039.stdout:694: done (1 left) 2026-02-25T08:44:42.177 INFO:tasks.rados.rados.0.trial039.stdout:695: write (excl) oid 13 current snap is 89 2026-02-25T08:44:42.177 INFO:tasks.rados.rados.0.trial039.stdout:695: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2026-02-25T08:44:42.180 INFO:tasks.rados.rados.0.trial039.stdout:695: writing trial03942481-13 from 463959 to 942404 tid 1 2026-02-25T08:44:42.186 INFO:tasks.rados.rados.0.trial039.stdout:695: writing trial03942481-13 from 1570667 to 2301775 tid 2 2026-02-25T08:44:42.191 INFO:tasks.rados.rados.0.trial039.stdout:695: writing trial03942481-13 from 3038372 to 3530805 tid 3 2026-02-25T08:44:42.191 INFO:tasks.rados.rados.0.trial039.stdout:696: delete oid 17 current snap is 89 2026-02-25T08:44:42.191 INFO:tasks.rados.rados.0.trial039.stdout:693: finishing write tid 1 to trial03942481-44 2026-02-25T08:44:42.197 INFO:tasks.rados.rados.0.trial039.stdout:693: finishing write tid 2 to trial03942481-44 2026-02-25T08:44:42.197 INFO:tasks.rados.rados.0.trial039.stdout:696: done (2 left) 2026-02-25T08:44:42.197 INFO:tasks.rados.rados.0.trial039.stdout:697: write oid 18 current snap is 89 2026-02-25T08:44:42.197 INFO:tasks.rados.rados.0.trial039.stdout:697: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2026-02-25T08:44:42.201 INFO:tasks.rados.rados.0.trial039.stdout:697: writing trial03942481-18 from 497573 to 1126516 tid 1 2026-02-25T08:44:42.206 INFO:tasks.rados.rados.0.trial039.stdout:697: writing trial03942481-18 from 1687631 to 2422408 tid 2 2026-02-25T08:44:42.211 INFO:tasks.rados.rados.0.trial039.stdout:697: writing trial03942481-18 from 3093218 to 3697573 tid 3 2026-02-25T08:44:42.212 INFO:tasks.rados.rados.0.trial039.stdout:698: read oid 42 snap -1 2026-02-25T08:44:42.212 INFO:tasks.rados.rados.0.trial039.stdout:698: expect (ObjNum 256 snap 71 seq_num 256) 2026-02-25T08:44:42.212 INFO:tasks.rados.rados.0.trial039.stdout:699: write oid 33 current snap is 89 2026-02-25T08:44:42.212 INFO:tasks.rados.rados.0.trial039.stdout:693: finishing write tid 3 to trial03942481-44 2026-02-25T08:44:42.212 INFO:tasks.rados.rados.0.trial039.stdout:699: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2026-02-25T08:44:42.216 INFO:tasks.rados.rados.0.trial039.stdout:699: writing trial03942481-33 from 514380 to 1218572 tid 1 2026-02-25T08:44:42.220 INFO:tasks.rados.rados.0.trial039.stdout:699: writing trial03942481-33 from 1629760 to 2108195 tid 2 2026-02-25T08:44:42.224 INFO:tasks.rados.rados.0.trial039.stdout:699: writing trial03942481-33 from 2604288 to 3046890 tid 3 2026-02-25T08:44:42.224 INFO:tasks.rados.rados.0.trial039.stdout:700: write (excl) oid 43 current snap is 89 2026-02-25T08:44:42.224 INFO:tasks.rados.rados.0.trial039.stdout:693: finishing write tid 4 to trial03942481-44 2026-02-25T08:44:42.224 INFO:tasks.rados.rados.0.trial039.stdout:700: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2026-02-25T08:44:42.229 INFO:tasks.rados.rados.0.trial039.stdout:700: writing trial03942481-43 from 531187 to 1310628 tid 1 2026-02-25T08:44:42.235 INFO:tasks.rados.rados.0.trial039.stdout:700: writing trial03942481-43 from 2088242 to 2826688 tid 2 2026-02-25T08:44:42.239 INFO:tasks.rados.rados.0.trial039.stdout:700: writing trial03942481-43 from 3431711 to 3731187 tid 3 2026-02-25T08:44:42.239 INFO:tasks.rados.rados.0.trial039.stdout:701: snap_create 2026-02-25T08:44:42.239 INFO:tasks.rados.rados.0.trial039.stdout:693: finishing write tid 5 to trial03942481-44 2026-02-25T08:44:42.239 INFO:tasks.rados.rados.0.trial039.stdout:693: oid 44 updating version 0 to 121 2026-02-25T08:44:42.239 INFO:tasks.rados.rados.0.trial039.stdout:693: oid 44 version 121 is already newer than 121 2026-02-25T08:44:42.239 INFO:tasks.rados.rados.0.trial039.stdout:693: oid 44 version 121 is already newer than 121 2026-02-25T08:44:42.239 INFO:tasks.rados.rados.0.trial039.stdout:693: oid 44 version 121 is already newer than 118 2026-02-25T08:44:42.239 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 121 (ObjNum 335 snap 89 seq_num 335) dirty exists 2026-02-25T08:44:42.239 INFO:tasks.rados.rados.0.trial039.stdout:693: left oid 44 (ObjNum 335 snap 89 seq_num 335) 2026-02-25T08:44:42.239 INFO:tasks.rados.rados.0.trial039.stdout:695: finishing write tid 1 to trial03942481-13 2026-02-25T08:44:42.240 INFO:tasks.rados.rados.0.trial039.stdout:698: expect (ObjNum 256 snap 71 seq_num 256) 2026-02-25T08:44:42.257 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1793577 2026-02-25T08:44:42.285 INFO:tasks.rados.rados.0.trial039.stdout:695: finishing write tid 2 to trial03942481-13 2026-02-25T08:44:42.285 INFO:tasks.rados.rados.0.trial039.stdout:695: finishing write tid 3 to trial03942481-13 2026-02-25T08:44:42.285 INFO:tasks.rados.rados.0.trial039.stdout:695: finishing write tid 4 to trial03942481-13 2026-02-25T08:44:42.285 INFO:tasks.rados.rados.0.trial039.stdout:695: finishing write tid 5 to trial03942481-13 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:695: oid 13 updating version 0 to 121 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:695: oid 13 updating version 121 to 124 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:695: oid 13 version 124 is already newer than 124 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:695: oid 13 version 124 is already newer than 124 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 13 v 124 (ObjNum 337 snap 89 seq_num 337) dirty exists 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:695: left oid 13 (ObjNum 337 snap 89 seq_num 337) 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:699: finishing write tid 1 to trial03942481-33 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:699: finishing write tid 2 to trial03942481-33 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:697: finishing write tid 1 to trial03942481-18 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:697: finishing write tid 2 to trial03942481-18 2026-02-25T08:44:42.286 INFO:tasks.rados.rados.0.trial039.stdout:699: finishing write tid 3 to trial03942481-33 2026-02-25T08:44:42.287 INFO:tasks.rados.rados.0.trial039.stdout:699: finishing write tid 4 to trial03942481-33 2026-02-25T08:44:42.287 INFO:tasks.rados.rados.0.trial039.stdout:697: finishing write tid 3 to trial03942481-18 2026-02-25T08:44:42.289 INFO:tasks.rados.rados.0.trial039.stdout:697: finishing write tid 4 to trial03942481-18 2026-02-25T08:44:42.290 INFO:tasks.rados.rados.0.trial039.stdout:699: finishing write tid 5 to trial03942481-33 2026-02-25T08:44:42.290 INFO:tasks.rados.rados.0.trial039.stdout:699: oid 33 updating version 0 to 109 2026-02-25T08:44:42.291 INFO:tasks.rados.rados.0.trial039.stdout:699: oid 33 version 109 is already newer than 109 2026-02-25T08:44:42.291 INFO:tasks.rados.rados.0.trial039.stdout:699: oid 33 version 109 is already newer than 109 2026-02-25T08:44:42.291 INFO:tasks.rados.rados.0.trial039.stdout:699: oid 33 version 109 is already newer than 109 2026-02-25T08:44:42.291 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 33 v 109 (ObjNum 340 snap 89 seq_num 340) dirty exists 2026-02-25T08:44:42.291 INFO:tasks.rados.rados.0.trial039.stdout:699: left oid 33 (ObjNum 340 snap 89 seq_num 340) 2026-02-25T08:44:42.291 INFO:tasks.rados.rados.0.trial039.stdout:697: finishing write tid 5 to trial03942481-18 2026-02-25T08:44:42.292 INFO:tasks.rados.rados.0.trial039.stdout:697: oid 18 updating version 0 to 129 2026-02-25T08:44:42.292 INFO:tasks.rados.rados.0.trial039.stdout:697: oid 18 version 129 is already newer than 129 2026-02-25T08:44:42.292 INFO:tasks.rados.rados.0.trial039.stdout:697: oid 18 version 129 is already newer than 129 2026-02-25T08:44:42.292 INFO:tasks.rados.rados.0.trial039.stdout:697: oid 18 version 129 is already newer than 129 2026-02-25T08:44:42.292 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 18 v 129 (ObjNum 339 snap 89 seq_num 339) dirty exists 2026-02-25T08:44:42.292 INFO:tasks.rados.rados.0.trial039.stdout:697: left oid 18 (ObjNum 339 snap 89 seq_num 339) 2026-02-25T08:44:42.311 INFO:tasks.rados.rados.0.trial039.stdout:700: finishing write tid 1 to trial03942481-43 2026-02-25T08:44:42.326 INFO:tasks.rados.rados.0.trial039.stdout:700: finishing write tid 2 to trial03942481-43 2026-02-25T08:44:42.334 INFO:tasks.rados.rados.0.trial039.stdout:700: finishing write tid 3 to trial03942481-43 2026-02-25T08:44:42.338 INFO:tasks.rados.rados.0.trial039.stdout:700: finishing write tid 4 to trial03942481-43 2026-02-25T08:44:42.340 INFO:tasks.rados.rados.0.trial039.stdout:700: finishing write tid 5 to trial03942481-43 2026-02-25T08:44:42.340 INFO:tasks.rados.rados.0.trial039.stdout:700: oid 43 updating version 0 to 114 2026-02-25T08:44:42.340 INFO:tasks.rados.rados.0.trial039.stdout:700: oid 43 version 114 is already newer than 114 2026-02-25T08:44:42.340 INFO:tasks.rados.rados.0.trial039.stdout:700: oid 43 version 114 is already newer than 111 2026-02-25T08:44:42.340 INFO:tasks.rados.rados.0.trial039.stdout:700: oid 43 version 114 is already newer than 114 2026-02-25T08:44:42.341 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 43 v 114 (ObjNum 341 snap 89 seq_num 341) dirty exists 2026-02-25T08:44:42.341 INFO:tasks.rados.rados.0.trial039.stdout:700: left oid 43 (ObjNum 341 snap 89 seq_num 341) 2026-02-25T08:44:42.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~187s 2026-02-25T08:44:43.078 INFO:tasks.rados.rados.0.trial039.stdout:693: done (6 left) 2026-02-25T08:44:43.078 INFO:tasks.rados.rados.0.trial039.stdout:695: done (5 left) 2026-02-25T08:44:43.078 INFO:tasks.rados.rados.0.trial039.stdout:697: done (4 left) 2026-02-25T08:44:43.078 INFO:tasks.rados.rados.0.trial039.stdout:698: done (3 left) 2026-02-25T08:44:43.078 INFO:tasks.rados.rados.0.trial039.stdout:699: done (2 left) 2026-02-25T08:44:43.079 INFO:tasks.rados.rados.0.trial039.stdout:700: done (1 left) 2026-02-25T08:44:43.079 INFO:tasks.rados.rados.0.trial039.stdout:701: done (0 left) 2026-02-25T08:44:43.079 INFO:tasks.rados.rados.0.trial039.stdout:702: snap_create 2026-02-25T08:44:43.364 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:43.442 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:43.438+0000 7fb1a50cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:43.444 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:43.442+0000 7fb1a50cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:43.473 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:43.489 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:44.078 INFO:tasks.rados.rados.0.trial039.stdout:702: done (0 left) 2026-02-25T08:44:44.079 INFO:tasks.rados.rados.0.trial039.stdout:703: write (excl) oid 2 current snap is 91 2026-02-25T08:44:44.079 INFO:tasks.rados.rados.0.trial039.stdout:703: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2026-02-25T08:44:44.082 INFO:tasks.rados.rados.0.trial039.stdout:703: writing trial03942481-2 from 547994 to 1002684 tid 1 2026-02-25T08:44:44.087 INFO:tasks.rados.rados.0.trial039.stdout:703: writing trial03942481-2 from 1746724 to 2228828 tid 2 2026-02-25T08:44:44.091 INFO:tasks.rados.rados.0.trial039.stdout:703: writing trial03942481-2 from 2659134 to 3124280 tid 3 2026-02-25T08:44:44.093 INFO:tasks.rados.rados.0.trial039.stdout:703: writing trial03942481-2 from 3605976 to 3747994 tid 4 2026-02-25T08:44:44.093 INFO:tasks.rados.rados.0.trial039.stdout:704: snap_create 2026-02-25T08:44:44.119 INFO:tasks.rados.rados.0.trial039.stdout:703: finishing write tid 1 to trial03942481-2 2026-02-25T08:44:44.136 INFO:tasks.rados.rados.0.trial039.stdout:703: finishing write tid 2 to trial03942481-2 2026-02-25T08:44:44.144 INFO:tasks.rados.rados.0.trial039.stdout:703: finishing write tid 3 to trial03942481-2 2026-02-25T08:44:44.152 INFO:tasks.rados.rados.0.trial039.stdout:703: finishing write tid 4 to trial03942481-2 2026-02-25T08:44:44.157 INFO:tasks.rados.rados.0.trial039.stdout:703: finishing write tid 5 to trial03942481-2 2026-02-25T08:44:44.159 INFO:tasks.rados.rados.0.trial039.stdout:703: finishing write tid 6 to trial03942481-2 2026-02-25T08:44:44.159 INFO:tasks.rados.rados.0.trial039.stdout:703: oid 2 updating version 0 to 127 2026-02-25T08:44:44.159 INFO:tasks.rados.rados.0.trial039.stdout:703: oid 2 version 127 is already newer than 127 2026-02-25T08:44:44.159 INFO:tasks.rados.rados.0.trial039.stdout:703: oid 2 version 127 is already newer than 127 2026-02-25T08:44:44.159 INFO:tasks.rados.rados.0.trial039.stdout:703: oid 2 version 127 is already newer than 127 2026-02-25T08:44:44.159 INFO:tasks.rados.rados.0.trial039.stdout:703: oid 2 version 127 is already newer than 123 2026-02-25T08:44:44.159 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 127 (ObjNum 342 snap 91 seq_num 342) dirty exists 2026-02-25T08:44:44.159 INFO:tasks.rados.rados.0.trial039.stdout:703: left oid 2 (ObjNum 342 snap 91 seq_num 342) 2026-02-25T08:44:45.079 INFO:tasks.rados.rados.0.trial039.stdout:703: done (1 left) 2026-02-25T08:44:45.080 INFO:tasks.rados.rados.0.trial039.stdout:704: done (0 left) 2026-02-25T08:44:45.080 INFO:tasks.rados.rados.0.trial039.stdout:705: read oid 26 snap 87 2026-02-25T08:44:45.080 INFO:tasks.rados.rados.0.trial039.stdout:705: expect deleted 2026-02-25T08:44:45.080 INFO:tasks.rados.rados.0.trial039.stdout:706: write oid 5 current snap is 92 2026-02-25T08:44:45.080 INFO:tasks.rados.rados.0.trial039.stdout:706: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2026-02-25T08:44:45.084 INFO:tasks.rados.rados.0.trial039.stdout:706: writing trial03942481-5 from 564801 to 1211093 tid 1 2026-02-25T08:44:45.090 INFO:tasks.rados.rados.0.trial039.stdout:706: writing trial03942481-5 from 1921559 to 2663674 tid 2 2026-02-25T08:44:45.094 INFO:tasks.rados.rados.0.trial039.stdout:706: writing trial03942481-5 from 3202910 to 3764801 tid 3 2026-02-25T08:44:45.094 INFO:tasks.rados.rados.0.trial039.stdout:707: snap_remove snap 88 2026-02-25T08:44:45.490 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:45.567 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:45.563+0000 7f89637e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:45.569 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:45.567+0000 7f89637e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:45.596 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:45.613 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:46.082 INFO:tasks.rados.rados.0.trial039.stdout:707: done (2 left) 2026-02-25T08:44:46.082 INFO:tasks.rados.rados.0.trial039.stdout:708: read oid 12 snap -1 2026-02-25T08:44:46.082 INFO:tasks.rados.rados.0.trial039.stdout:708: expect deleted 2026-02-25T08:44:46.082 INFO:tasks.rados.rados.0.trial039.stdout:709: write (excl) oid 2 current snap is 92 2026-02-25T08:44:46.082 INFO:tasks.rados.rados.0.trial039.stdout:709: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2026-02-25T08:44:46.086 INFO:tasks.rados.rados.0.trial039.stdout:709: writing trial03942481-2 from 581608 to 1303149 tid 1 2026-02-25T08:44:46.090 INFO:tasks.rados.rados.0.trial039.stdout:709: writing trial03942481-2 from 1863688 to 2349461 tid 2 2026-02-25T08:44:46.096 INFO:tasks.rados.rados.0.trial039.stdout:709: writing trial03942481-2 from 3113980 to 3718023 tid 3 2026-02-25T08:44:46.096 INFO:tasks.rados.rados.0.trial039.stdout:710: read oid 23 snap -1 2026-02-25T08:44:46.096 INFO:tasks.rados.rados.0.trial039.stdout:710: expect (ObjNum 299 snap 80 seq_num 299) 2026-02-25T08:44:46.097 INFO:tasks.rados.rados.0.trial039.stdout:711: read oid 13 snap -1 2026-02-25T08:44:46.097 INFO:tasks.rados.rados.0.trial039.stdout:711: expect (ObjNum 337 snap 89 seq_num 337) 2026-02-25T08:44:46.097 INFO:tasks.rados.rados.0.trial039.stdout:712: write oid 41 current snap is 92 2026-02-25T08:44:46.097 INFO:tasks.rados.rados.0.trial039.stdout:712: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2026-02-25T08:44:46.101 INFO:tasks.rados.rados.0.trial039.stdout:712: writing trial03942481-41 from 598415 to 1395205 tid 1 2026-02-25T08:44:46.106 INFO:tasks.rados.rados.0.trial039.stdout:712: writing trial03942481-41 from 1922170 to 2667954 tid 2 2026-02-25T08:44:46.110 INFO:tasks.rados.rados.0.trial039.stdout:712: writing trial03942481-41 from 3141403 to 3556718 tid 3 2026-02-25T08:44:46.111 INFO:tasks.rados.rados.0.trial039.stdout:713: read oid 36 snap -1 2026-02-25T08:44:46.111 INFO:tasks.rados.rados.0.trial039.stdout:713: expect (ObjNum 301 snap 81 seq_num 301) 2026-02-25T08:44:46.111 INFO:tasks.rados.rados.0.trial039.stdout:714: delete oid 16 current snap is 92 2026-02-25T08:44:46.111 INFO:tasks.rados.rados.0.trial039.stdout:706: finishing write tid 1 to trial03942481-5 2026-02-25T08:44:46.111 INFO:tasks.rados.rados.0.trial039.stdout:706: finishing write tid 2 to trial03942481-5 2026-02-25T08:44:46.111 INFO:tasks.rados.rados.0.trial039.stdout:706: finishing write tid 3 to trial03942481-5 2026-02-25T08:44:46.111 INFO:tasks.rados.rados.0.trial039.stdout:706: finishing write tid 4 to trial03942481-5 2026-02-25T08:44:46.111 INFO:tasks.rados.rados.0.trial039.stdout:706: finishing write tid 5 to trial03942481-5 2026-02-25T08:44:46.111 INFO:tasks.rados.rados.0.trial039.stdout:706: oid 5 updating version 0 to 108 2026-02-25T08:44:46.112 INFO:tasks.rados.rados.0.trial039.stdout:706: oid 5 version 108 is already newer than 108 2026-02-25T08:44:46.112 INFO:tasks.rados.rados.0.trial039.stdout:706: oid 5 version 108 is already newer than 108 2026-02-25T08:44:46.112 INFO:tasks.rados.rados.0.trial039.stdout:706: oid 5 version 108 is already newer than 108 2026-02-25T08:44:46.112 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 5 v 108 (ObjNum 343 snap 92 seq_num 343) dirty exists 2026-02-25T08:44:46.112 INFO:tasks.rados.rados.0.trial039.stdout:706: left oid 5 (ObjNum 343 snap 92 seq_num 343) 2026-02-25T08:44:46.122 INFO:tasks.rados.rados.0.trial039.stdout:710: expect (ObjNum 299 snap 80 seq_num 299) 2026-02-25T08:44:46.123 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:46.161 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 539706 2026-02-25T08:44:46.184 INFO:tasks.rados.rados.0.trial039.stdout:705: done (8 left) 2026-02-25T08:44:46.184 INFO:tasks.rados.rados.0.trial039.stdout:706: done (7 left) 2026-02-25T08:44:46.184 INFO:tasks.rados.rados.0.trial039.stdout:708: done (6 left) 2026-02-25T08:44:46.184 INFO:tasks.rados.rados.0.trial039.stdout:710: done (5 left) 2026-02-25T08:44:46.185 INFO:tasks.rados.rados.0.trial039.stdout:714: done (4 left) 2026-02-25T08:44:46.185 INFO:tasks.rados.rados.0.trial039.stdout:715: write oid 9 current snap is 92 2026-02-25T08:44:46.185 INFO:tasks.rados.rados.0.trial039.stdout:711: expect (ObjNum 337 snap 89 seq_num 337) 2026-02-25T08:44:46.193 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1577757 2026-02-25T08:44:46.264 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:46.309 INFO:tasks.rados.rados.0.trial039.stdout:712: finishing write tid 1 to trial03942481-41 2026-02-25T08:44:46.309 INFO:tasks.rados.rados.0.trial039.stdout:709: finishing write tid 1 to trial03942481-2 2026-02-25T08:44:46.309 INFO:tasks.rados.rados.0.trial039.stdout:712: finishing write tid 2 to trial03942481-41 2026-02-25T08:44:46.309 INFO:tasks.rados.rados.0.trial039.stdout:715: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2026-02-25T08:44:46.314 INFO:tasks.rados.rados.0.trial039.stdout:715: writing trial03942481-9 from 632029 to 1179317 tid 1 2026-02-25T08:44:46.318 INFO:tasks.rados.rados.0.trial039.stdout:715: writing trial03942481-9 from 1639134 to 2388587 tid 2 2026-02-25T08:44:46.322 INFO:tasks.rados.rados.0.trial039.stdout:715: writing trial03942481-9 from 2796249 to 3234108 tid 3 2026-02-25T08:44:46.324 INFO:tasks.rados.rados.0.trial039.stdout:715: writing trial03942481-9 from 3653524 to 3832029 tid 4 2026-02-25T08:44:46.324 INFO:tasks.rados.rados.0.trial039.stdout:711: done (4 left) 2026-02-25T08:44:46.325 INFO:tasks.rados.rados.0.trial039.stdout:716: delete oid 23 current snap is 92 2026-02-25T08:44:46.325 INFO:tasks.rados.rados.0.trial039.stdout:712: finishing write tid 3 to trial03942481-41 2026-02-25T08:44:46.325 INFO:tasks.rados.rados.0.trial039.stdout:713: expect (ObjNum 301 snap 81 seq_num 301) 2026-02-25T08:44:46.325 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:46.416 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 3033320 2026-02-25T08:44:46.416 INFO:tasks.rados.rados.0.trial039.stdout:712: finishing write tid 4 to trial03942481-41 2026-02-25T08:44:46.416 INFO:tasks.rados.rados.0.trial039.stdout:712: finishing write tid 5 to trial03942481-41 2026-02-25T08:44:46.416 INFO:tasks.rados.rados.0.trial039.stdout:712: oid 41 updating version 0 to 61 2026-02-25T08:44:46.416 INFO:tasks.rados.rados.0.trial039.stdout:712: oid 41 version 61 is already newer than 61 2026-02-25T08:44:46.416 INFO:tasks.rados.rados.0.trial039.stdout:712: oid 41 version 61 is already newer than 61 2026-02-25T08:44:46.416 INFO:tasks.rados.rados.0.trial039.stdout:712: oid 41 version 61 is already newer than 61 2026-02-25T08:44:46.416 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 41 v 61 (ObjNum 345 snap 92 seq_num 345) dirty exists 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:712: left oid 41 (ObjNum 345 snap 92 seq_num 345) 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:709: finishing write tid 2 to trial03942481-2 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:709: finishing write tid 3 to trial03942481-2 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:709: finishing write tid 4 to trial03942481-2 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:709: finishing write tid 5 to trial03942481-2 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:709: oid 2 updating version 0 to 132 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:709: oid 2 version 132 is already newer than 132 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:709: oid 2 version 132 is already newer than 129 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:709: oid 2 version 132 is already newer than 132 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 132 (ObjNum 344 snap 92 seq_num 344) dirty exists 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:709: left oid 2 (ObjNum 344 snap 92 seq_num 344) 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:715: finishing write tid 1 to trial03942481-9 2026-02-25T08:44:46.417 INFO:tasks.rados.rados.0.trial039.stdout:715: finishing write tid 2 to trial03942481-9 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:715: finishing write tid 3 to trial03942481-9 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:715: finishing write tid 4 to trial03942481-9 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:715: finishing write tid 5 to trial03942481-9 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:715: finishing write tid 6 to trial03942481-9 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:715: oid 9 updating version 0 to 34 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:715: oid 9 version 34 is already newer than 34 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:715: oid 9 version 34 is already newer than 34 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:715: oid 9 version 34 is already newer than 34 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:715: oid 9 version 34 is already newer than 34 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 9 v 34 (ObjNum 347 snap 92 seq_num 347) dirty exists 2026-02-25T08:44:46.418 INFO:tasks.rados.rados.0.trial039.stdout:715: left oid 9 (ObjNum 347 snap 92 seq_num 347) 2026-02-25T08:44:46.419 INFO:tasks.rados.rados.0.trial039.stdout:709: done (4 left) 2026-02-25T08:44:46.419 INFO:tasks.rados.rados.0.trial039.stdout:712: done (3 left) 2026-02-25T08:44:46.419 INFO:tasks.rados.rados.0.trial039.stdout:713: done (2 left) 2026-02-25T08:44:46.419 INFO:tasks.rados.rados.0.trial039.stdout:715: done (1 left) 2026-02-25T08:44:46.419 INFO:tasks.rados.rados.0.trial039.stdout:716: done (0 left) 2026-02-25T08:44:46.419 INFO:tasks.rados.rados.0.trial039.stdout:717: snap_create 2026-02-25T08:44:46.667 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:46.739 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:46.742 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:46.743 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:47.088 INFO:tasks.rados.rados.0.trial039.stdout:717: done (0 left) 2026-02-25T08:44:47.088 INFO:tasks.rados.rados.0.trial039.stdout:718: write (excl) oid 41 current snap is 93 2026-02-25T08:44:47.088 INFO:tasks.rados.rados.0.trial039.stdout:718: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2026-02-25T08:44:47.093 INFO:tasks.rados.rados.0.trial039.stdout:718: writing trial03942481-41 from 665643 to 1363429 tid 1 2026-02-25T08:44:47.099 INFO:tasks.rados.rados.0.trial039.stdout:718: writing trial03942481-41 from 2039745 to 2792867 tid 2 2026-02-25T08:44:47.104 INFO:tasks.rados.rados.0.trial039.stdout:718: writing trial03942481-41 from 3534742 to 3865643 tid 3 2026-02-25T08:44:47.104 INFO:tasks.rados.rados.0.trial039.stdout:719: read oid 17 snap -1 2026-02-25T08:44:47.104 INFO:tasks.rados.rados.0.trial039.stdout:719: expect deleted 2026-02-25T08:44:47.104 INFO:tasks.rados.rados.0.trial039.stdout:720: snap_create 2026-02-25T08:44:47.122 INFO:tasks.rados.rados.0.trial039.stdout:718: finishing write tid 1 to trial03942481-41 2026-02-25T08:44:47.137 INFO:tasks.rados.rados.0.trial039.stdout:718: finishing write tid 2 to trial03942481-41 2026-02-25T08:44:47.145 INFO:tasks.rados.rados.0.trial039.stdout:718: finishing write tid 3 to trial03942481-41 2026-02-25T08:44:47.149 INFO:tasks.rados.rados.0.trial039.stdout:718: finishing write tid 4 to trial03942481-41 2026-02-25T08:44:47.151 INFO:tasks.rados.rados.0.trial039.stdout:718: finishing write tid 5 to trial03942481-41 2026-02-25T08:44:47.151 INFO:tasks.rados.rados.0.trial039.stdout:718: oid 41 updating version 0 to 66 2026-02-25T08:44:47.152 INFO:tasks.rados.rados.0.trial039.stdout:718: oid 41 version 66 is already newer than 66 2026-02-25T08:44:47.152 INFO:tasks.rados.rados.0.trial039.stdout:718: oid 41 version 66 is already newer than 66 2026-02-25T08:44:47.152 INFO:tasks.rados.rados.0.trial039.stdout:718: oid 41 version 66 is already newer than 63 2026-02-25T08:44:47.152 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 41 v 66 (ObjNum 349 snap 93 seq_num 349) dirty exists 2026-02-25T08:44:47.152 INFO:tasks.rados.rados.0.trial039.stdout:718: left oid 41 (ObjNum 349 snap 93 seq_num 349) 2026-02-25T08:44:47.614 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:47.688 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:47.684+0000 7f6be525a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:47.690 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:47.688+0000 7f6be525a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:47.716 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:47.732 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:47.733 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:47.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~193s 2026-02-25T08:44:48.087 INFO:tasks.rados.rados.0.trial039.stdout:718: done (2 left) 2026-02-25T08:44:48.088 INFO:tasks.rados.rados.0.trial039.stdout:719: done (1 left) 2026-02-25T08:44:48.088 INFO:tasks.rados.rados.0.trial039.stdout:720: done (0 left) 2026-02-25T08:44:48.088 INFO:tasks.rados.rados.0.trial039.stdout:721: write oid 30 current snap is 94 2026-02-25T08:44:48.088 INFO:tasks.rados.rados.0.trial039.stdout:721: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2026-02-25T08:44:48.092 INFO:tasks.rados.rados.0.trial039.stdout:721: writing trial03942481-30 from 682450 to 1171838 tid 1 2026-02-25T08:44:48.096 INFO:tasks.rados.rados.0.trial039.stdout:721: writing trial03942481-30 from 1814580 to 2311360 tid 2 2026-02-25T08:44:48.102 INFO:tasks.rados.rados.0.trial039.stdout:721: writing trial03942481-30 from 2878518 to 3666546 tid 3 2026-02-25T08:44:48.102 INFO:tasks.rados.rados.0.trial039.stdout:722: write oid 46 current snap is 94 2026-02-25T08:44:48.103 INFO:tasks.rados.rados.0.trial039.stdout:722: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2026-02-25T08:44:48.107 INFO:tasks.rados.rados.0.trial039.stdout:722: writing trial03942481-46 from 699257 to 1263894 tid 1 2026-02-25T08:44:48.112 INFO:tasks.rados.rados.0.trial039.stdout:722: writing trial03942481-46 from 1873062 to 2513500 tid 2 2026-02-25T08:44:48.117 INFO:tasks.rados.rados.0.trial039.stdout:722: writing trial03942481-46 from 3305941 to 3899257 tid 3 2026-02-25T08:44:48.117 INFO:tasks.rados.rados.0.trial039.stdout:723: snap_remove snap 92 2026-02-25T08:44:49.092 INFO:tasks.rados.rados.0.trial039.stdout:723: done (2 left) 2026-02-25T08:44:49.093 INFO:tasks.rados.rados.0.trial039.stdout:724: read oid 41 snap 91 2026-02-25T08:44:49.093 INFO:tasks.rados.rados.0.trial039.stdout:724: expect deleted 2026-02-25T08:44:49.093 INFO:tasks.rados.rados.0.trial039.stdout:725: write oid 7 current snap is 94 2026-02-25T08:44:49.093 INFO:tasks.rados.rados.0.trial039.stdout:722: finishing write tid 1 to trial03942481-46 2026-02-25T08:44:49.093 INFO:tasks.rados.rados.0.trial039.stdout:725: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2026-02-25T08:44:49.097 INFO:tasks.rados.rados.0.trial039.stdout:725: writing trial03942481-7 from 716064 to 1355950 tid 1 2026-02-25T08:44:49.101 INFO:tasks.rados.rados.0.trial039.stdout:725: writing trial03942481-7 from 1815191 to 2315640 tid 2 2026-02-25T08:44:49.106 INFO:tasks.rados.rados.0.trial039.stdout:725: writing trial03942481-7 from 2817011 to 3343936 tid 3 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:726: snap_create 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:721: finishing write tid 1 to trial03942481-30 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:722: finishing write tid 2 to trial03942481-46 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:721: finishing write tid 2 to trial03942481-30 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:722: finishing write tid 3 to trial03942481-46 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:721: finishing write tid 3 to trial03942481-30 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:722: finishing write tid 4 to trial03942481-46 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:722: finishing write tid 5 to trial03942481-46 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:722: oid 46 updating version 0 to 115 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:722: oid 46 version 115 is already newer than 115 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:722: oid 46 version 115 is already newer than 115 2026-02-25T08:44:49.107 INFO:tasks.rados.rados.0.trial039.stdout:722: oid 46 version 115 is already newer than 115 2026-02-25T08:44:49.108 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 46 v 115 (ObjNum 351 snap 94 seq_num 351) dirty exists 2026-02-25T08:44:49.108 INFO:tasks.rados.rados.0.trial039.stdout:722: left oid 46 (ObjNum 351 snap 94 seq_num 351) 2026-02-25T08:44:49.108 INFO:tasks.rados.rados.0.trial039.stdout:721: finishing write tid 4 to trial03942481-30 2026-02-25T08:44:49.108 INFO:tasks.rados.rados.0.trial039.stdout:721: finishing write tid 5 to trial03942481-30 2026-02-25T08:44:49.108 INFO:tasks.rados.rados.0.trial039.stdout:721: oid 30 updating version 0 to 91 2026-02-25T08:44:49.108 INFO:tasks.rados.rados.0.trial039.stdout:721: oid 30 version 91 is already newer than 91 2026-02-25T08:44:49.108 INFO:tasks.rados.rados.0.trial039.stdout:721: oid 30 version 91 is already newer than 91 2026-02-25T08:44:49.108 INFO:tasks.rados.rados.0.trial039.stdout:721: oid 30 version 91 is already newer than 91 2026-02-25T08:44:49.108 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 30 v 91 (ObjNum 350 snap 94 seq_num 350) dirty exists 2026-02-25T08:44:49.108 INFO:tasks.rados.rados.0.trial039.stdout:721: left oid 30 (ObjNum 350 snap 94 seq_num 350) 2026-02-25T08:44:49.131 INFO:tasks.rados.rados.0.trial039.stdout:725: finishing write tid 1 to trial03942481-7 2026-02-25T08:44:49.139 INFO:tasks.rados.rados.0.trial039.stdout:725: finishing write tid 2 to trial03942481-7 2026-02-25T08:44:49.146 INFO:tasks.rados.rados.0.trial039.stdout:725: finishing write tid 3 to trial03942481-7 2026-02-25T08:44:49.151 INFO:tasks.rados.rados.0.trial039.stdout:725: finishing write tid 4 to trial03942481-7 2026-02-25T08:44:49.153 INFO:tasks.rados.rados.0.trial039.stdout:725: finishing write tid 5 to trial03942481-7 2026-02-25T08:44:49.153 INFO:tasks.rados.rados.0.trial039.stdout:725: oid 7 updating version 0 to 136 2026-02-25T08:44:49.153 INFO:tasks.rados.rados.0.trial039.stdout:725: oid 7 version 136 is already newer than 136 2026-02-25T08:44:49.153 INFO:tasks.rados.rados.0.trial039.stdout:725: oid 7 version 136 is already newer than 136 2026-02-25T08:44:49.154 INFO:tasks.rados.rados.0.trial039.stdout:725: oid 7 version 136 is already newer than 136 2026-02-25T08:44:49.154 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 7 v 136 (ObjNum 352 snap 94 seq_num 352) dirty exists 2026-02-25T08:44:49.154 INFO:tasks.rados.rados.0.trial039.stdout:725: left oid 7 (ObjNum 352 snap 94 seq_num 352) 2026-02-25T08:44:49.735 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:49.812 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:49.809+0000 7f21081fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:49.815 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:49.812+0000 7f21081fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:49.840 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:49.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:50.097 INFO:tasks.rados.rados.0.trial039.stdout:721: done (4 left) 2026-02-25T08:44:50.098 INFO:tasks.rados.rados.0.trial039.stdout:722: done (3 left) 2026-02-25T08:44:50.098 INFO:tasks.rados.rados.0.trial039.stdout:724: done (2 left) 2026-02-25T08:44:50.098 INFO:tasks.rados.rados.0.trial039.stdout:725: done (1 left) 2026-02-25T08:44:50.098 INFO:tasks.rados.rados.0.trial039.stdout:726: done (0 left) 2026-02-25T08:44:50.098 INFO:tasks.rados.rados.0.trial039.stdout:727: write (excl) oid 7 current snap is 95 2026-02-25T08:44:50.098 INFO:tasks.rados.rados.0.trial039.stdout:727: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2026-02-25T08:44:50.103 INFO:tasks.rados.rados.0.trial039.stdout:727: writing trial03942481-7 from 732871 to 1448006 tid 1 2026-02-25T08:44:50.108 INFO:tasks.rados.rados.0.trial039.stdout:727: writing trial03942481-7 from 1873673 to 2517780 tid 2 2026-02-25T08:44:50.113 INFO:tasks.rados.rados.0.trial039.stdout:727: writing trial03942481-7 from 3244434 to 3932871 tid 3 2026-02-25T08:44:50.114 INFO:tasks.rados.rados.0.trial039.stdout:728: snap_remove snap 85 2026-02-25T08:44:51.100 INFO:tasks.rados.rados.0.trial039.stdout:728: done (1 left) 2026-02-25T08:44:51.100 INFO:tasks.rados.rados.0.trial039.stdout:729: snap_remove snap 93 2026-02-25T08:44:51.744 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:51.818 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:51.821 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:51.821 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:51.859 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:51.938 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:51.934+0000 7f47e1ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:51.940 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:51.938+0000 7f47e1ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:51.966 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:51.984 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:51.984 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:52.101 INFO:tasks.rados.rados.0.trial039.stdout:729: done (1 left) 2026-02-25T08:44:52.101 INFO:tasks.rados.rados.0.trial039.stdout:730: snap_create 2026-02-25T08:44:52.101 INFO:tasks.rados.rados.0.trial039.stdout:727: finishing write tid 1 to trial03942481-7 2026-02-25T08:44:52.101 INFO:tasks.rados.rados.0.trial039.stdout:727: finishing write tid 2 to trial03942481-7 2026-02-25T08:44:52.101 INFO:tasks.rados.rados.0.trial039.stdout:727: finishing write tid 3 to trial03942481-7 2026-02-25T08:44:52.101 INFO:tasks.rados.rados.0.trial039.stdout:727: finishing write tid 4 to trial03942481-7 2026-02-25T08:44:52.101 INFO:tasks.rados.rados.0.trial039.stdout:727: finishing write tid 5 to trial03942481-7 2026-02-25T08:44:52.102 INFO:tasks.rados.rados.0.trial039.stdout:727: oid 7 updating version 0 to 141 2026-02-25T08:44:52.102 INFO:tasks.rados.rados.0.trial039.stdout:727: oid 7 version 141 is already newer than 141 2026-02-25T08:44:52.102 INFO:tasks.rados.rados.0.trial039.stdout:727: oid 7 version 141 is already newer than 141 2026-02-25T08:44:52.102 INFO:tasks.rados.rados.0.trial039.stdout:727: oid 7 version 141 is already newer than 138 2026-02-25T08:44:52.102 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 7 v 141 (ObjNum 353 snap 95 seq_num 353) dirty exists 2026-02-25T08:44:52.102 INFO:tasks.rados.rados.0.trial039.stdout:727: left oid 7 (ObjNum 353 snap 95 seq_num 353) 2026-02-25T08:44:53.079 INFO:tasks.rados.rados.0.trial039.stdout:727: done (1 left) 2026-02-25T08:44:53.080 INFO:tasks.rados.rados.0.trial039.stdout:730: done (0 left) 2026-02-25T08:44:53.080 INFO:tasks.rados.rados.0.trial039.stdout:731: read oid 26 snap -1 2026-02-25T08:44:53.080 INFO:tasks.rados.rados.0.trial039.stdout:731: expect deleted 2026-02-25T08:44:53.080 INFO:tasks.rados.rados.0.trial039.stdout:732: write oid 44 current snap is 96 2026-02-25T08:44:53.080 INFO:tasks.rados.rados.0.trial039.stdout:732: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2026-02-25T08:44:53.086 INFO:tasks.rados.rados.0.trial039.stdout:732: writing trial03942481-44 from 749678 to 1540062 tid 1 2026-02-25T08:44:53.092 INFO:tasks.rados.rados.0.trial039.stdout:732: writing trial03942481-44 from 2332155 to 2836273 tid 2 2026-02-25T08:44:53.096 INFO:tasks.rados.rados.0.trial039.stdout:732: writing trial03942481-44 from 3271857 to 3821326 tid 3 2026-02-25T08:44:53.097 INFO:tasks.rados.rados.0.trial039.stdout:733: read oid 5 snap -1 2026-02-25T08:44:53.097 INFO:tasks.rados.rados.0.trial039.stdout:733: expect (ObjNum 343 snap 92 seq_num 343) 2026-02-25T08:44:53.097 INFO:tasks.rados.rados.0.trial039.stdout:734: read oid 45 snap -1 2026-02-25T08:44:53.097 INFO:tasks.rados.rados.0.trial039.stdout:734: expect deleted 2026-02-25T08:44:53.097 INFO:tasks.rados.rados.0.trial039.stdout:735: snap_create 2026-02-25T08:44:53.116 INFO:tasks.rados.rados.0.trial039.stdout:733: expect (ObjNum 343 snap 92 seq_num 343) 2026-02-25T08:44:53.147 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:53.211 INFO:tasks.rados.rados.0.trial039.stdout:732: finishing write tid 1 to trial03942481-44 2026-02-25T08:44:53.211 INFO:tasks.rados.rados.0.trial039.stdout:732: finishing write tid 2 to trial03942481-44 2026-02-25T08:44:53.211 INFO:tasks.rados.rados.0.trial039.stdout:732: finishing write tid 3 to trial03942481-44 2026-02-25T08:44:53.211 INFO:tasks.rados.rados.0.trial039.stdout:732: finishing write tid 4 to trial03942481-44 2026-02-25T08:44:53.211 INFO:tasks.rados.rados.0.trial039.stdout:732: finishing write tid 5 to trial03942481-44 2026-02-25T08:44:53.211 INFO:tasks.rados.rados.0.trial039.stdout:732: oid 44 updating version 0 to 146 2026-02-25T08:44:53.211 INFO:tasks.rados.rados.0.trial039.stdout:732: oid 44 version 146 is already newer than 146 2026-02-25T08:44:53.211 INFO:tasks.rados.rados.0.trial039.stdout:732: oid 44 version 146 is already newer than 146 2026-02-25T08:44:53.211 INFO:tasks.rados.rados.0.trial039.stdout:732: oid 44 version 146 is already newer than 146 2026-02-25T08:44:53.212 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 146 (ObjNum 354 snap 96 seq_num 354) dirty exists 2026-02-25T08:44:53.212 INFO:tasks.rados.rados.0.trial039.stdout:732: left oid 44 (ObjNum 354 snap 96 seq_num 354) 2026-02-25T08:44:53.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~198s 2026-02-25T08:44:53.985 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:54.062 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:54.059+0000 7f18a34cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:54.065 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:54.062+0000 7f18a34cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:54.078 INFO:tasks.rados.rados.0.trial039.stdout:731: done (4 left) 2026-02-25T08:44:54.079 INFO:tasks.rados.rados.0.trial039.stdout:732: done (3 left) 2026-02-25T08:44:54.079 INFO:tasks.rados.rados.0.trial039.stdout:733: done (2 left) 2026-02-25T08:44:54.079 INFO:tasks.rados.rados.0.trial039.stdout:734: done (1 left) 2026-02-25T08:44:54.079 INFO:tasks.rados.rados.0.trial039.stdout:735: done (0 left) 2026-02-25T08:44:54.079 INFO:tasks.rados.rados.0.trial039.stdout:736: write oid 28 current snap is 97 2026-02-25T08:44:54.079 INFO:tasks.rados.rados.0.trial039.stdout:736: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2026-02-25T08:44:54.083 INFO:tasks.rados.rados.0.trial039.stdout:736: writing trial03942481-28 from 766485 to 1232118 tid 1 2026-02-25T08:44:54.089 INFO:tasks.rados.rados.0.trial039.stdout:736: writing trial03942481-28 from 1990637 to 2638413 tid 2 2026-02-25T08:44:54.095 INFO:tasks.rados.rados.0.trial039.stdout:736: writing trial03942481-28 from 3299280 to 3966485 tid 3 2026-02-25T08:44:54.095 INFO:tasks.rados.rados.0.trial039.stdout:737: read oid 8 snap -1 2026-02-25T08:44:54.095 INFO:tasks.rados.rados.0.trial039.stdout:737: expect (ObjNum 333 snap 87 seq_num 333) 2026-02-25T08:44:54.095 INFO:tasks.rados.rados.0.trial039.stdout:738: write (excl) oid 35 current snap is 97 2026-02-25T08:44:54.095 INFO:tasks.rados.rados.0.trial039.stdout:738: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2026-02-25T08:44:54.099 INFO:tasks.rados.rados.0.trial039.stdout:738: writing trial03942481-35 from 783292 to 1324174 tid 1 2026-02-25T08:44:54.103 INFO:tasks.rados.rados.0.trial039.stdout:738: writing trial03942481-35 from 1932766 to 2440553 tid 2 2026-02-25T08:44:54.107 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:54.109 INFO:tasks.rados.rados.0.trial039.stdout:738: writing trial03942481-35 from 3210350 to 3782363 tid 3 2026-02-25T08:44:54.109 INFO:tasks.rados.rados.0.trial039.stdout:739: write oid 50 current snap is 97 2026-02-25T08:44:54.109 INFO:tasks.rados.rados.0.trial039.stdout:739: seq_num 357 ranges {400099=616131,1591248=408851} 2026-02-25T08:44:54.112 INFO:tasks.rados.rados.0.trial039.stdout:739: writing trial03942481-50 from 400099 to 1016230 tid 1 2026-02-25T08:44:54.116 INFO:tasks.rados.rados.0.trial039.stdout:739: writing trial03942481-50 from 1591248 to 2000099 tid 2 2026-02-25T08:44:54.116 INFO:tasks.rados.rados.0.trial039.stdout:740: write oid 26 current snap is 97 2026-02-25T08:44:54.116 INFO:tasks.rados.rados.0.trial039.stdout:740: seq_num 358 ranges {416906=407733,1366083=511456} 2026-02-25T08:44:54.119 INFO:tasks.rados.rados.0.trial039.stdout:740: writing trial03942481-26 from 416906 to 824639 tid 1 2026-02-25T08:44:54.123 INFO:tasks.rados.rados.0.trial039.stdout:740: writing trial03942481-26 from 1366083 to 1877539 tid 2 2026-02-25T08:44:54.123 INFO:tasks.rados.rados.0.trial039.stdout:741: write oid 42 current snap is 97 2026-02-25T08:44:54.124 INFO:tasks.rados.rados.0.trial039.stdout:741: seq_num 359 ranges {433713=482982,1424565=609148} 2026-02-25T08:44:54.124 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:54.125 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:54.127 INFO:tasks.rados.rados.0.trial039.stdout:741: writing trial03942481-42 from 433713 to 916695 tid 1 2026-02-25T08:44:54.131 INFO:tasks.rados.rados.0.trial039.stdout:741: writing trial03942481-42 from 1424565 to 2033713 tid 2 2026-02-25T08:44:54.131 INFO:tasks.rados.rados.0.trial039.stdout:742: delete oid 44 current snap is 97 2026-02-25T08:44:54.150 INFO:tasks.rados.rados.0.trial039.stdout:739: finishing write tid 1 to trial03942481-50 2026-02-25T08:44:54.165 INFO:tasks.rados.rados.0.trial039.stdout:739: finishing write tid 2 to trial03942481-50 2026-02-25T08:44:54.169 INFO:tasks.rados.rados.0.trial039.stdout:737: expect (ObjNum 333 snap 87 seq_num 333) 2026-02-25T08:44:54.187 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2569267 2026-02-25T08:44:54.204 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1018266 2026-02-25T08:44:54.236 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 220930 2026-02-25T08:44:54.277 INFO:tasks.rados.rados.0.trial039.stdout:739: finishing write tid 3 to trial03942481-50 2026-02-25T08:44:54.277 INFO:tasks.rados.rados.0.trial039.stdout:739: finishing write tid 4 to trial03942481-50 2026-02-25T08:44:54.277 INFO:tasks.rados.rados.0.trial039.stdout:739: oid 50 updating version 0 to 144 2026-02-25T08:44:54.277 INFO:tasks.rados.rados.0.trial039.stdout:739: oid 50 version 144 is already newer than 144 2026-02-25T08:44:54.277 INFO:tasks.rados.rados.0.trial039.stdout:739: oid 50 version 144 is already newer than 144 2026-02-25T08:44:54.277 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 50 v 144 (ObjNum 357 snap 97 seq_num 357) dirty exists 2026-02-25T08:44:54.277 INFO:tasks.rados.rados.0.trial039.stdout:739: left oid 50 (ObjNum 357 snap 97 seq_num 357) 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:738: finishing write tid 1 to trial03942481-35 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:737: done (6 left) 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:739: done (5 left) 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:742: done (4 left) 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:743: delete oid 33 current snap is 97 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:740: finishing write tid 1 to trial03942481-26 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:740: finishing write tid 2 to trial03942481-26 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:741: finishing write tid 1 to trial03942481-42 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:740: finishing write tid 3 to trial03942481-26 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:740: finishing write tid 4 to trial03942481-26 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:740: oid 26 updating version 0 to 69 2026-02-25T08:44:54.278 INFO:tasks.rados.rados.0.trial039.stdout:740: oid 26 version 69 is already newer than 69 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:740: oid 26 version 69 is already newer than 69 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 26 v 69 (ObjNum 358 snap 97 seq_num 358) dirty exists 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:740: left oid 26 (ObjNum 358 snap 97 seq_num 358) 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:741: finishing write tid 2 to trial03942481-42 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:738: finishing write tid 2 to trial03942481-35 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:738: finishing write tid 3 to trial03942481-35 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:741: finishing write tid 3 to trial03942481-42 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:741: finishing write tid 4 to trial03942481-42 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:741: oid 42 updating version 0 to 267 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:741: oid 42 version 267 is already newer than 267 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:741: oid 42 version 267 is already newer than 267 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 42 v 267 (ObjNum 359 snap 97 seq_num 359) dirty exists 2026-02-25T08:44:54.279 INFO:tasks.rados.rados.0.trial039.stdout:741: left oid 42 (ObjNum 359 snap 97 seq_num 359) 2026-02-25T08:44:54.280 INFO:tasks.rados.rados.0.trial039.stdout:738: finishing write tid 4 to trial03942481-35 2026-02-25T08:44:54.280 INFO:tasks.rados.rados.0.trial039.stdout:738: finishing write tid 5 to trial03942481-35 2026-02-25T08:44:54.280 INFO:tasks.rados.rados.0.trial039.stdout:738: oid 35 updating version 0 to 106 2026-02-25T08:44:54.280 INFO:tasks.rados.rados.0.trial039.stdout:738: oid 35 updating version 106 to 109 2026-02-25T08:44:54.280 INFO:tasks.rados.rados.0.trial039.stdout:738: oid 35 version 109 is already newer than 109 2026-02-25T08:44:54.280 INFO:tasks.rados.rados.0.trial039.stdout:738: oid 35 version 109 is already newer than 109 2026-02-25T08:44:54.280 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 35 v 109 (ObjNum 356 snap 97 seq_num 356) dirty exists 2026-02-25T08:44:54.280 INFO:tasks.rados.rados.0.trial039.stdout:738: left oid 35 (ObjNum 356 snap 97 seq_num 356) 2026-02-25T08:44:54.280 INFO:tasks.rados.rados.0.trial039.stdout:736: finishing write tid 1 to trial03942481-28 2026-02-25T08:44:54.280 INFO:tasks.rados.rados.0.trial039.stdout:736: finishing write tid 2 to trial03942481-28 2026-02-25T08:44:54.283 INFO:tasks.rados.rados.0.trial039.stdout:736: finishing write tid 3 to trial03942481-28 2026-02-25T08:44:54.289 INFO:tasks.rados.rados.0.trial039.stdout:736: finishing write tid 4 to trial03942481-28 2026-02-25T08:44:54.293 INFO:tasks.rados.rados.0.trial039.stdout:736: finishing write tid 5 to trial03942481-28 2026-02-25T08:44:54.293 INFO:tasks.rados.rados.0.trial039.stdout:736: oid 28 updating version 0 to 119 2026-02-25T08:44:54.293 INFO:tasks.rados.rados.0.trial039.stdout:736: oid 28 version 119 is already newer than 119 2026-02-25T08:44:54.293 INFO:tasks.rados.rados.0.trial039.stdout:736: oid 28 version 119 is already newer than 119 2026-02-25T08:44:54.293 INFO:tasks.rados.rados.0.trial039.stdout:736: oid 28 version 119 is already newer than 119 2026-02-25T08:44:54.293 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 28 v 119 (ObjNum 355 snap 97 seq_num 355) dirty exists 2026-02-25T08:44:54.293 INFO:tasks.rados.rados.0.trial039.stdout:736: left oid 28 (ObjNum 355 snap 97 seq_num 355) 2026-02-25T08:44:54.313 INFO:tasks.rados.rados.0.trial039.stdout:736: done (4 left) 2026-02-25T08:44:54.314 INFO:tasks.rados.rados.0.trial039.stdout:738: done (3 left) 2026-02-25T08:44:54.314 INFO:tasks.rados.rados.0.trial039.stdout:740: done (2 left) 2026-02-25T08:44:54.314 INFO:tasks.rados.rados.0.trial039.stdout:741: done (1 left) 2026-02-25T08:44:54.314 INFO:tasks.rados.rados.0.trial039.stdout:743: done (0 left) 2026-02-25T08:44:54.314 INFO:tasks.rados.rados.0.trial039.stdout:744: delete oid 46 current snap is 97 2026-02-25T08:44:54.349 INFO:tasks.rados.rados.0.trial039.stdout:744: done (0 left) 2026-02-25T08:44:54.350 INFO:tasks.rados.rados.0.trial039.stdout:745: delete oid 47 current snap is 97 2026-02-25T08:44:54.381 INFO:tasks.rados.rados.0.trial039.stdout:745: done (0 left) 2026-02-25T08:44:54.381 INFO:tasks.rados.rados.0.trial039.stdout:746: delete oid 44 current snap is 97 2026-02-25T08:44:54.383 INFO:tasks.rados.rados.0.trial039.stdout:746: done (0 left) 2026-02-25T08:44:54.383 INFO:tasks.rados.rados.0.trial039.stdout:747: write (excl) oid 14 current snap is 97 2026-02-25T08:44:54.383 INFO:tasks.rados.rados.0.trial039.stdout:747: seq_num 365 ranges {534555=650829,1659104=475451} 2026-02-25T08:44:54.388 INFO:tasks.rados.rados.0.trial039.stdout:747: writing trial03942481-14 from 534555 to 1185384 tid 1 2026-02-25T08:44:54.392 INFO:tasks.rados.rados.0.trial039.stdout:747: writing trial03942481-14 from 1659104 to 2134555 tid 2 2026-02-25T08:44:54.392 INFO:tasks.rados.rados.0.trial039.stdout:748: snap_create 2026-02-25T08:44:54.404 INFO:tasks.rados.rados.0.trial039.stdout:747: finishing write tid 1 to trial03942481-14 2026-02-25T08:44:54.419 INFO:tasks.rados.rados.0.trial039.stdout:747: finishing write tid 2 to trial03942481-14 2026-02-25T08:44:54.424 INFO:tasks.rados.rados.0.trial039.stdout:747: finishing write tid 3 to trial03942481-14 2026-02-25T08:44:54.426 INFO:tasks.rados.rados.0.trial039.stdout:747: finishing write tid 4 to trial03942481-14 2026-02-25T08:44:54.426 INFO:tasks.rados.rados.0.trial039.stdout:747: oid 14 updating version 0 to 147 2026-02-25T08:44:54.426 INFO:tasks.rados.rados.0.trial039.stdout:747: oid 14 version 147 is already newer than 147 2026-02-25T08:44:54.426 INFO:tasks.rados.rados.0.trial039.stdout:747: oid 14 version 147 is already newer than 145 2026-02-25T08:44:54.426 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 14 v 147 (ObjNum 365 snap 97 seq_num 365) dirty exists 2026-02-25T08:44:54.426 INFO:tasks.rados.rados.0.trial039.stdout:747: left oid 14 (ObjNum 365 snap 97 seq_num 365) 2026-02-25T08:44:55.108 INFO:tasks.rados.rados.0.trial039.stdout:747: done (1 left) 2026-02-25T08:44:55.108 INFO:tasks.rados.rados.0.trial039.stdout:748: done (0 left) 2026-02-25T08:44:55.109 INFO:tasks.rados.rados.0.trial039.stdout:749: delete oid 1 current snap is 98 2026-02-25T08:44:55.151 INFO:tasks.rados.rados.0.trial039.stdout:749: done (0 left) 2026-02-25T08:44:55.151 INFO:tasks.rados.rados.0.trial039.stdout:750: write oid 25 current snap is 98 2026-02-25T08:44:55.152 INFO:tasks.rados.rados.0.trial039.stdout:750: seq_num 367 ranges {568169=401327,1376068=669790} 2026-02-25T08:44:55.155 INFO:tasks.rados.rados.0.trial039.stdout:750: writing trial03942481-25 from 568169 to 969496 tid 1 2026-02-25T08:44:55.160 INFO:tasks.rados.rados.0.trial039.stdout:750: writing trial03942481-25 from 1376068 to 2045858 tid 2 2026-02-25T08:44:55.160 INFO:tasks.rados.rados.0.trial039.stdout:751: write oid 37 current snap is 98 2026-02-25T08:44:55.160 INFO:tasks.rados.rados.0.trial039.stdout:751: seq_num 368 ranges {584976=476576,1834550=350426} 2026-02-25T08:44:55.164 INFO:tasks.rados.rados.0.trial039.stdout:751: writing trial03942481-37 from 584976 to 1061552 tid 1 2026-02-25T08:44:55.168 INFO:tasks.rados.rados.0.trial039.stdout:751: writing trial03942481-37 from 1834550 to 2184976 tid 2 2026-02-25T08:44:55.168 INFO:tasks.rados.rados.0.trial039.stdout:752: snap_create 2026-02-25T08:44:55.187 INFO:tasks.rados.rados.0.trial039.stdout:751: finishing write tid 1 to trial03942481-37 2026-02-25T08:44:55.194 INFO:tasks.rados.rados.0.trial039.stdout:750: finishing write tid 1 to trial03942481-25 2026-02-25T08:44:55.195 INFO:tasks.rados.rados.0.trial039.stdout:751: finishing write tid 2 to trial03942481-37 2026-02-25T08:44:55.202 INFO:tasks.rados.rados.0.trial039.stdout:750: finishing write tid 2 to trial03942481-25 2026-02-25T08:44:55.207 INFO:tasks.rados.rados.0.trial039.stdout:750: finishing write tid 3 to trial03942481-25 2026-02-25T08:44:55.208 INFO:tasks.rados.rados.0.trial039.stdout:751: finishing write tid 3 to trial03942481-37 2026-02-25T08:44:55.209 INFO:tasks.rados.rados.0.trial039.stdout:750: finishing write tid 4 to trial03942481-25 2026-02-25T08:44:55.209 INFO:tasks.rados.rados.0.trial039.stdout:750: oid 25 updating version 0 to 19 2026-02-25T08:44:55.210 INFO:tasks.rados.rados.0.trial039.stdout:750: oid 25 version 19 is already newer than 19 2026-02-25T08:44:55.210 INFO:tasks.rados.rados.0.trial039.stdout:750: oid 25 version 19 is already newer than 19 2026-02-25T08:44:55.210 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 25 v 19 (ObjNum 367 snap 98 seq_num 367) dirty exists 2026-02-25T08:44:55.210 INFO:tasks.rados.rados.0.trial039.stdout:750: left oid 25 (ObjNum 367 snap 98 seq_num 367) 2026-02-25T08:44:55.210 INFO:tasks.rados.rados.0.trial039.stdout:751: finishing write tid 4 to trial03942481-37 2026-02-25T08:44:55.211 INFO:tasks.rados.rados.0.trial039.stdout:751: oid 37 updating version 0 to 151 2026-02-25T08:44:55.211 INFO:tasks.rados.rados.0.trial039.stdout:751: oid 37 version 151 is already newer than 151 2026-02-25T08:44:55.211 INFO:tasks.rados.rados.0.trial039.stdout:751: oid 37 version 151 is already newer than 151 2026-02-25T08:44:55.211 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 37 v 151 (ObjNum 368 snap 98 seq_num 368) dirty exists 2026-02-25T08:44:55.211 INFO:tasks.rados.rados.0.trial039.stdout:751: left oid 37 (ObjNum 368 snap 98 seq_num 368) 2026-02-25T08:44:56.114 INFO:tasks.rados.rados.0.trial039.stdout:750: done (2 left) 2026-02-25T08:44:56.114 INFO:tasks.rados.rados.0.trial039.stdout:751: done (1 left) 2026-02-25T08:44:56.114 INFO:tasks.rados.rados.0.trial039.stdout:752: done (0 left) 2026-02-25T08:44:56.115 INFO:tasks.rados.rados.0.trial039.stdout:753: write oid 23 current snap is 99 2026-02-25T08:44:56.115 INFO:tasks.rados.rados.0.trial039.stdout:753: seq_num 369 ranges {601783=551825,1776679=425104} 2026-02-25T08:44:56.119 INFO:tasks.rados.rados.0.trial039.stdout:753: writing trial03942481-23 from 601783 to 1153608 tid 1 2026-02-25T08:44:56.123 INFO:tasks.rados.rados.0.trial039.stdout:753: writing trial03942481-23 from 1776679 to 2201783 tid 2 2026-02-25T08:44:56.123 INFO:tasks.rados.rados.0.trial039.stdout:754: snap_create 2026-02-25T08:44:56.125 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:56.148 INFO:tasks.rados.rados.0.trial039.stdout:753: finishing write tid 1 to trial03942481-23 2026-02-25T08:44:56.155 INFO:tasks.rados.rados.0.trial039.stdout:753: finishing write tid 2 to trial03942481-23 2026-02-25T08:44:56.159 INFO:tasks.rados.rados.0.trial039.stdout:753: finishing write tid 3 to trial03942481-23 2026-02-25T08:44:56.168 INFO:tasks.rados.rados.0.trial039.stdout:753: finishing write tid 4 to trial03942481-23 2026-02-25T08:44:56.168 INFO:tasks.rados.rados.0.trial039.stdout:753: oid 23 updating version 0 to 128 2026-02-25T08:44:56.169 INFO:tasks.rados.rados.0.trial039.stdout:753: oid 23 version 128 is already newer than 128 2026-02-25T08:44:56.169 INFO:tasks.rados.rados.0.trial039.stdout:753: oid 23 version 128 is already newer than 128 2026-02-25T08:44:56.169 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 23 v 128 (ObjNum 369 snap 99 seq_num 369) dirty exists 2026-02-25T08:44:56.169 INFO:tasks.rados.rados.0.trial039.stdout:753: left oid 23 (ObjNum 369 snap 99 seq_num 369) 2026-02-25T08:44:56.203 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:56.201+0000 7f6069011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:56.206 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:56.204+0000 7f6069011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:56.233 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:56.254 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:56.822 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:44:56.896 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:56.899 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:44:57.116 INFO:tasks.rados.rados.0.trial039.stdout:753: done (1 left) 2026-02-25T08:44:57.116 INFO:tasks.rados.rados.0.trial039.stdout:754: done (0 left) 2026-02-25T08:44:57.116 INFO:tasks.rados.rados.0.trial039.stdout:755: read oid 17 snap -1 2026-02-25T08:44:57.116 INFO:tasks.rados.rados.0.trial039.stdout:755: expect deleted 2026-02-25T08:44:57.117 INFO:tasks.rados.rados.0.trial039.stdout:756: read oid 50 snap -1 2026-02-25T08:44:57.117 INFO:tasks.rados.rados.0.trial039.stdout:756: expect (ObjNum 357 snap 97 seq_num 357) 2026-02-25T08:44:57.117 INFO:tasks.rados.rados.0.trial039.stdout:757: snap_remove snap 95 2026-02-25T08:44:58.121 INFO:tasks.rados.rados.0.trial039.stdout:757: done (2 left) 2026-02-25T08:44:58.121 INFO:tasks.rados.rados.0.trial039.stdout:758: write oid 19 current snap is 100 2026-02-25T08:44:58.121 INFO:tasks.rados.rados.0.trial039.stdout:758: seq_num 370 ranges {618590=627074,1835161=383429} 2026-02-25T08:44:58.125 INFO:tasks.rados.rados.0.trial039.stdout:758: writing trial03942481-19 from 618590 to 1245664 tid 1 2026-02-25T08:44:58.129 INFO:tasks.rados.rados.0.trial039.stdout:758: writing trial03942481-19 from 1835161 to 2218590 tid 2 2026-02-25T08:44:58.129 INFO:tasks.rados.rados.0.trial039.stdout:759: snap_remove snap 90 2026-02-25T08:44:58.255 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:58.330 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:58.328+0000 7f4a4759c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:58.333 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:44:58.331+0000 7f4a4759c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:58.359 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:44:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:58.376 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:58.979 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~204s 2026-02-25T08:44:59.127 INFO:tasks.rados.rados.0.trial039.stdout:759: done (3 left) 2026-02-25T08:44:59.128 INFO:tasks.rados.rados.0.trial039.stdout:760: read oid 33 snap -1 2026-02-25T08:44:59.128 INFO:tasks.rados.rados.0.trial039.stdout:760: expect deleted 2026-02-25T08:44:59.128 INFO:tasks.rados.rados.0.trial039.stdout:761: delete oid 18 current snap is 100 2026-02-25T08:44:59.128 INFO:tasks.rados.rados.0.trial039.stdout:756: expect (ObjNum 357 snap 97 seq_num 357) 2026-02-25T08:44:59.128 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:59.171 INFO:tasks.rados.rados.0.trial039.stdout:755: done (4 left) 2026-02-25T08:44:59.171 INFO:tasks.rados.rados.0.trial039.stdout:756: done (3 left) 2026-02-25T08:44:59.171 INFO:tasks.rados.rados.0.trial039.stdout:761: done (2 left) 2026-02-25T08:44:59.171 INFO:tasks.rados.rados.0.trial039.stdout:762: write oid 12 current snap is 100 2026-02-25T08:44:59.171 INFO:tasks.rados.rados.0.trial039.stdout:762: seq_num 372 ranges {652204=777572,1952125=300079} 2026-02-25T08:44:59.176 INFO:tasks.rados.rados.0.trial039.stdout:762: writing trial03942481-12 from 652204 to 1429776 tid 1 2026-02-25T08:44:59.180 INFO:tasks.rados.rados.0.trial039.stdout:762: writing trial03942481-12 from 1952125 to 2252204 tid 2 2026-02-25T08:44:59.180 INFO:tasks.rados.rados.0.trial039.stdout:763: read oid 35 snap -1 2026-02-25T08:44:59.180 INFO:tasks.rados.rados.0.trial039.stdout:763: expect (ObjNum 356 snap 97 seq_num 356) 2026-02-25T08:44:59.180 INFO:tasks.rados.rados.0.trial039.stdout:764: write (excl) oid 4 current snap is 100 2026-02-25T08:44:59.180 INFO:tasks.rados.rados.0.trial039.stdout:764: seq_num 373 ranges {669011=569174,2010607=258404} 2026-02-25T08:44:59.184 INFO:tasks.rados.rados.0.trial039.stdout:764: writing trial03942481-4 from 669011 to 1238185 tid 1 2026-02-25T08:44:59.189 INFO:tasks.rados.rados.0.trial039.stdout:764: writing trial03942481-4 from 2010607 to 2269011 tid 2 2026-02-25T08:44:59.189 INFO:tasks.rados.rados.0.trial039.stdout:765: write oid 34 current snap is 100 2026-02-25T08:44:59.189 INFO:tasks.rados.rados.0.trial039.stdout:758: finishing write tid 1 to trial03942481-19 2026-02-25T08:44:59.189 INFO:tasks.rados.rados.0.trial039.stdout:765: seq_num 374 ranges {685818=644423,2069089=216729} 2026-02-25T08:44:59.193 INFO:tasks.rados.rados.0.trial039.stdout:765: writing trial03942481-34 from 685818 to 1330241 tid 1 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:765: writing trial03942481-34 from 2069089 to 2285818 tid 2 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:766: snap_create 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:758: finishing write tid 2 to trial03942481-19 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:758: finishing write tid 3 to trial03942481-19 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:758: finishing write tid 4 to trial03942481-19 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:758: oid 19 updating version 0 to 270 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:758: oid 19 version 270 is already newer than 270 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:758: oid 19 version 270 is already newer than 270 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 19 v 270 (ObjNum 370 snap 100 seq_num 370) dirty exists 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:758: left oid 19 (ObjNum 370 snap 100 seq_num 370) 2026-02-25T08:44:59.198 INFO:tasks.rados.rados.0.trial039.stdout:763: expect (ObjNum 356 snap 97 seq_num 356) 2026-02-25T08:44:59.227 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:44:59.304 INFO:tasks.rados.rados.0.trial039.stdout:762: finishing write tid 1 to trial03942481-12 2026-02-25T08:44:59.304 INFO:tasks.rados.rados.0.trial039.stdout:762: finishing write tid 2 to trial03942481-12 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:762: finishing write tid 3 to trial03942481-12 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:762: finishing write tid 4 to trial03942481-12 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:762: oid 12 updating version 0 to 52 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:762: oid 12 version 52 is already newer than 52 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:762: oid 12 version 52 is already newer than 52 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 12 v 52 (ObjNum 372 snap 100 seq_num 372) dirty exists 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:762: left oid 12 (ObjNum 372 snap 100 seq_num 372) 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:765: finishing write tid 1 to trial03942481-34 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:764: finishing write tid 1 to trial03942481-4 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:765: finishing write tid 2 to trial03942481-34 2026-02-25T08:44:59.305 INFO:tasks.rados.rados.0.trial039.stdout:764: finishing write tid 2 to trial03942481-4 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:764: finishing write tid 3 to trial03942481-4 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:764: finishing write tid 4 to trial03942481-4 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:764: oid 4 updating version 0 to 134 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:764: oid 4 version 134 is already newer than 134 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:764: oid 4 version 134 is already newer than 132 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 134 (ObjNum 373 snap 100 seq_num 373) dirty exists 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:764: left oid 4 (ObjNum 373 snap 100 seq_num 373) 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:765: finishing write tid 3 to trial03942481-34 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:765: finishing write tid 4 to trial03942481-34 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:765: oid 34 updating version 0 to 151 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:765: oid 34 version 151 is already newer than 151 2026-02-25T08:44:59.306 INFO:tasks.rados.rados.0.trial039.stdout:765: oid 34 version 151 is already newer than 151 2026-02-25T08:44:59.307 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 151 (ObjNum 374 snap 100 seq_num 374) dirty exists 2026-02-25T08:44:59.307 INFO:tasks.rados.rados.0.trial039.stdout:765: left oid 34 (ObjNum 374 snap 100 seq_num 374) 2026-02-25T08:45:00.130 INFO:tasks.rados.rados.0.trial039.stdout:758: done (6 left) 2026-02-25T08:45:00.130 INFO:tasks.rados.rados.0.trial039.stdout:760: done (5 left) 2026-02-25T08:45:00.130 INFO:tasks.rados.rados.0.trial039.stdout:762: done (4 left) 2026-02-25T08:45:00.131 INFO:tasks.rados.rados.0.trial039.stdout:763: done (3 left) 2026-02-25T08:45:00.131 INFO:tasks.rados.rados.0.trial039.stdout:764: done (2 left) 2026-02-25T08:45:00.131 INFO:tasks.rados.rados.0.trial039.stdout:765: done (1 left) 2026-02-25T08:45:00.131 INFO:tasks.rados.rados.0.trial039.stdout:766: done (0 left) 2026-02-25T08:45:00.131 INFO:tasks.rados.rados.0.trial039.stdout:767: read oid 14 snap 97 2026-02-25T08:45:00.131 INFO:tasks.rados.rados.0.trial039.stdout:767: expect (ObjNum 365 snap 97 seq_num 365) 2026-02-25T08:45:00.131 INFO:tasks.rados.rados.0.trial039.stdout:768: write oid 48 current snap is 101 2026-02-25T08:45:00.131 INFO:tasks.rados.rados.0.trial039.stdout:768: seq_num 375 ranges {702625=719672,2127571=175054} 2026-02-25T08:45:00.135 INFO:tasks.rados.rados.0.trial039.stdout:768: writing trial03942481-48 from 702625 to 1422297 tid 1 2026-02-25T08:45:00.139 INFO:tasks.rados.rados.0.trial039.stdout:768: writing trial03942481-48 from 2127571 to 2302625 tid 2 2026-02-25T08:45:00.139 INFO:tasks.rados.rados.0.trial039.stdout:769: write (excl) oid 50 current snap is 101 2026-02-25T08:45:00.139 INFO:tasks.rados.rados.0.trial039.stdout:769: seq_num 376 ranges {719432=794921,2186053=133379} 2026-02-25T08:45:00.144 INFO:tasks.rados.rados.0.trial039.stdout:769: writing trial03942481-50 from 719432 to 1514353 tid 1 2026-02-25T08:45:00.147 INFO:tasks.rados.rados.0.trial039.stdout:769: writing trial03942481-50 from 2186053 to 2319432 tid 2 2026-02-25T08:45:00.147 INFO:tasks.rados.rados.0.trial039.stdout:770: write (excl) oid 34 current snap is 101 2026-02-25T08:45:00.147 INFO:tasks.rados.rados.0.trial039.stdout:770: seq_num 377 ranges {736239=470170,1728182=571782} 2026-02-25T08:45:00.152 INFO:tasks.rados.rados.0.trial039.stdout:770: writing trial03942481-34 from 736239 to 1206409 tid 1 2026-02-25T08:45:00.156 INFO:tasks.rados.rados.0.trial039.stdout:770: writing trial03942481-34 from 1728182 to 2299964 tid 2 2026-02-25T08:45:00.157 INFO:tasks.rados.rados.0.trial039.stdout:771: read oid 23 snap -1 2026-02-25T08:45:00.157 INFO:tasks.rados.rados.0.trial039.stdout:771: expect (ObjNum 369 snap 99 seq_num 369) 2026-02-25T08:45:00.157 INFO:tasks.rados.rados.0.trial039.stdout:772: read oid 39 snap -1 2026-02-25T08:45:00.157 INFO:tasks.rados.rados.0.trial039.stdout:772: expect (ObjNum 313 snap 82 seq_num 313) 2026-02-25T08:45:00.157 INFO:tasks.rados.rados.0.trial039.stdout:773: write oid 40 current snap is 101 2026-02-25T08:45:00.157 INFO:tasks.rados.rados.0.trial039.stdout:773: seq_num 378 ranges {753046=545419,1786664=431793} 2026-02-25T08:45:00.161 INFO:tasks.rados.rados.0.trial039.stdout:773: writing trial03942481-40 from 753046 to 1298465 tid 1 2026-02-25T08:45:00.165 INFO:tasks.rados.rados.0.trial039.stdout:773: writing trial03942481-40 from 1786664 to 2218457 tid 2 2026-02-25T08:45:00.165 INFO:tasks.rados.rados.0.trial039.stdout:774: write (excl) oid 11 current snap is 101 2026-02-25T08:45:00.165 INFO:tasks.rados.rados.0.trial039.stdout:774: seq_num 379 ranges {769853=620668,1845146=524707} 2026-02-25T08:45:00.170 INFO:tasks.rados.rados.0.trial039.stdout:774: writing trial03942481-11 from 769853 to 1390521 tid 1 2026-02-25T08:45:00.174 INFO:tasks.rados.rados.0.trial039.stdout:774: writing trial03942481-11 from 1845146 to 2369853 tid 2 2026-02-25T08:45:00.174 INFO:tasks.rados.rados.0.trial039.stdout:775: read oid 13 snap -1 2026-02-25T08:45:00.174 INFO:tasks.rados.rados.0.trial039.stdout:775: expect (ObjNum 337 snap 89 seq_num 337) 2026-02-25T08:45:00.175 INFO:tasks.rados.rados.0.trial039.stdout:776: read oid 6 snap -1 2026-02-25T08:45:00.175 INFO:tasks.rados.rados.0.trial039.stdout:776: expect (ObjNum 302 snap 81 seq_num 302) 2026-02-25T08:45:00.175 INFO:tasks.rados.rados.0.trial039.stdout:777: write (excl) oid 32 current snap is 101 2026-02-25T08:45:00.175 INFO:tasks.rados.rados.0.trial039.stdout:777: seq_num 380 ranges {786660=695917,1903628=435462} 2026-02-25T08:45:00.179 INFO:tasks.rados.rados.0.trial039.stdout:777: writing trial03942481-32 from 786660 to 1482577 tid 1 2026-02-25T08:45:00.183 INFO:tasks.rados.rados.0.trial039.stdout:777: writing trial03942481-32 from 1903628 to 2339090 tid 2 2026-02-25T08:45:00.183 INFO:tasks.rados.rados.0.trial039.stdout:778: snap_create 2026-02-25T08:45:00.183 INFO:tasks.rados.rados.0.trial039.stdout:767: expect (ObjNum 365 snap 97 seq_num 365) 2026-02-25T08:45:00.219 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 128251 2026-02-25T08:45:00.235 INFO:tasks.rados.rados.0.trial039.stdout:772: expect (ObjNum 313 snap 82 seq_num 313) 2026-02-25T08:45:00.236 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:00.263 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:00.290 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:00.317 INFO:tasks.rados.rados.0.trial039.stdout:768: finishing write tid 1 to trial03942481-48 2026-02-25T08:45:00.318 INFO:tasks.rados.rados.0.trial039.stdout:768: finishing write tid 2 to trial03942481-48 2026-02-25T08:45:00.318 INFO:tasks.rados.rados.0.trial039.stdout:771: expect (ObjNum 369 snap 99 seq_num 369) 2026-02-25T08:45:00.318 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:00.359 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1153192 2026-02-25T08:45:00.368 INFO:tasks.rados.rados.0.trial039.stdout:776: expect (ObjNum 302 snap 81 seq_num 302) 2026-02-25T08:45:00.377 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:00.410 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:00.436 INFO:tasks.rados.rados.0.trial039.stdout:769: finishing write tid 1 to trial03942481-50 2026-02-25T08:45:00.436 INFO:tasks.rados.rados.0.trial039.stdout:769: finishing write tid 2 to trial03942481-50 2026-02-25T08:45:00.436 INFO:tasks.rados.rados.0.trial039.stdout:775: expect (ObjNum 337 snap 89 seq_num 337) 2026-02-25T08:45:00.454 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:00.451+0000 7fe2aa10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:00.457 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:00.455+0000 7fe2aa10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:00.471 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:00.484 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:00.499 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:00.515 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 231304 2026-02-25T08:45:00.558 INFO:tasks.rados.rados.0.trial039.stdout:768: finishing write tid 3 to trial03942481-48 2026-02-25T08:45:00.558 INFO:tasks.rados.rados.0.trial039.stdout:768: finishing write tid 4 to trial03942481-48 2026-02-25T08:45:00.558 INFO:tasks.rados.rados.0.trial039.stdout:768: oid 48 updating version 0 to 42 2026-02-25T08:45:00.558 INFO:tasks.rados.rados.0.trial039.stdout:768: oid 48 version 42 is already newer than 42 2026-02-25T08:45:00.558 INFO:tasks.rados.rados.0.trial039.stdout:768: oid 48 version 42 is already newer than 42 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 48 v 42 (ObjNum 375 snap 101 seq_num 375) dirty exists 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:768: left oid 48 (ObjNum 375 snap 101 seq_num 375) 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:774: finishing write tid 1 to trial03942481-11 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:777: finishing write tid 1 to trial03942481-32 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:769: finishing write tid 3 to trial03942481-50 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:769: finishing write tid 4 to trial03942481-50 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:769: oid 50 updating version 0 to 155 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:769: oid 50 version 155 is already newer than 155 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:769: oid 50 version 155 is already newer than 153 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 50 v 155 (ObjNum 376 snap 101 seq_num 376) dirty exists 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:769: left oid 50 (ObjNum 376 snap 101 seq_num 376) 2026-02-25T08:45:00.559 INFO:tasks.rados.rados.0.trial039.stdout:774: finishing write tid 2 to trial03942481-11 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:777: finishing write tid 2 to trial03942481-32 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:774: finishing write tid 3 to trial03942481-11 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:774: finishing write tid 4 to trial03942481-11 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:774: oid 11 updating version 0 to 24 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:774: oid 11 version 24 is already newer than 22 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:774: oid 11 version 24 is already newer than 24 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 11 v 24 (ObjNum 379 snap 101 seq_num 379) dirty exists 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:774: left oid 11 (ObjNum 379 snap 101 seq_num 379) 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:777: finishing write tid 3 to trial03942481-32 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:777: finishing write tid 4 to trial03942481-32 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:777: oid 32 updating version 0 to 95 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:777: oid 32 version 95 is already newer than 93 2026-02-25T08:45:00.560 INFO:tasks.rados.rados.0.trial039.stdout:777: oid 32 version 95 is already newer than 95 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 32 v 95 (ObjNum 380 snap 101 seq_num 380) dirty exists 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:777: left oid 32 (ObjNum 380 snap 101 seq_num 380) 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:770: finishing write tid 1 to trial03942481-34 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:770: finishing write tid 2 to trial03942481-34 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:770: finishing write tid 3 to trial03942481-34 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:770: finishing write tid 4 to trial03942481-34 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:770: oid 34 updating version 0 to 157 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:770: oid 34 updating version 157 to 159 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:770: oid 34 version 159 is already newer than 159 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 159 (ObjNum 377 snap 101 seq_num 377) dirty exists 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:770: left oid 34 (ObjNum 377 snap 101 seq_num 377) 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:773: finishing write tid 1 to trial03942481-40 2026-02-25T08:45:00.561 INFO:tasks.rados.rados.0.trial039.stdout:773: finishing write tid 2 to trial03942481-40 2026-02-25T08:45:00.562 INFO:tasks.rados.rados.0.trial039.stdout:773: finishing write tid 3 to trial03942481-40 2026-02-25T08:45:00.562 INFO:tasks.rados.rados.0.trial039.stdout:773: finishing write tid 4 to trial03942481-40 2026-02-25T08:45:00.562 INFO:tasks.rados.rados.0.trial039.stdout:773: oid 40 updating version 0 to 163 2026-02-25T08:45:00.562 INFO:tasks.rados.rados.0.trial039.stdout:773: oid 40 version 163 is already newer than 163 2026-02-25T08:45:00.562 INFO:tasks.rados.rados.0.trial039.stdout:773: oid 40 version 163 is already newer than 163 2026-02-25T08:45:00.562 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 40 v 163 (ObjNum 378 snap 101 seq_num 378) dirty exists 2026-02-25T08:45:00.562 INFO:tasks.rados.rados.0.trial039.stdout:773: left oid 40 (ObjNum 378 snap 101 seq_num 378) 2026-02-25T08:45:01.136 INFO:tasks.rados.rados.0.trial039.stdout:767: done (11 left) 2026-02-25T08:45:01.137 INFO:tasks.rados.rados.0.trial039.stdout:768: done (10 left) 2026-02-25T08:45:01.137 INFO:tasks.rados.rados.0.trial039.stdout:769: done (9 left) 2026-02-25T08:45:01.137 INFO:tasks.rados.rados.0.trial039.stdout:770: done (8 left) 2026-02-25T08:45:01.137 INFO:tasks.rados.rados.0.trial039.stdout:771: done (7 left) 2026-02-25T08:45:01.137 INFO:tasks.rados.rados.0.trial039.stdout:772: done (6 left) 2026-02-25T08:45:01.137 INFO:tasks.rados.rados.0.trial039.stdout:773: done (5 left) 2026-02-25T08:45:01.137 INFO:tasks.rados.rados.0.trial039.stdout:774: done (4 left) 2026-02-25T08:45:01.137 INFO:tasks.rados.rados.0.trial039.stdout:775: done (3 left) 2026-02-25T08:45:01.137 INFO:tasks.rados.rados.0.trial039.stdout:776: done (2 left) 2026-02-25T08:45:01.137 INFO:tasks.rados.rados.0.trial039.stdout:777: done (1 left) 2026-02-25T08:45:01.138 INFO:tasks.rados.rados.0.trial039.stdout:778: done (0 left) 2026-02-25T08:45:01.138 INFO:tasks.rados.rados.0.trial039.stdout:779: snap_create 2026-02-25T08:45:01.900 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:01.972 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:01.976 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:02.138 INFO:tasks.rados.rados.0.trial039.stdout:779: done (0 left) 2026-02-25T08:45:02.138 INFO:tasks.rados.rados.0.trial039.stdout:780: read oid 24 snap 98 2026-02-25T08:45:02.138 INFO:tasks.rados.rados.0.trial039.stdout:780: expect deleted 2026-02-25T08:45:02.138 INFO:tasks.rados.rados.0.trial039.stdout:781: write (excl) oid 50 current snap is 103 2026-02-25T08:45:02.138 INFO:tasks.rados.rados.0.trial039.stdout:781: seq_num 381 ranges {403467=487519,1562110=579120} 2026-02-25T08:45:02.141 INFO:tasks.rados.rados.0.trial039.stdout:781: writing trial03942481-50 from 403467 to 890986 tid 1 2026-02-25T08:45:02.147 INFO:tasks.rados.rados.0.trial039.stdout:781: writing trial03942481-50 from 1562110 to 2141230 tid 2 2026-02-25T08:45:02.147 INFO:tasks.rados.rados.0.trial039.stdout:782: read oid 36 snap -1 2026-02-25T08:45:02.148 INFO:tasks.rados.rados.0.trial039.stdout:782: expect (ObjNum 301 snap 81 seq_num 301) 2026-02-25T08:45:02.148 INFO:tasks.rados.rados.0.trial039.stdout:783: read oid 16 snap -1 2026-02-25T08:45:02.148 INFO:tasks.rados.rados.0.trial039.stdout:783: expect deleted 2026-02-25T08:45:02.148 INFO:tasks.rados.rados.0.trial039.stdout:784: read oid 30 snap 94 2026-02-25T08:45:02.148 INFO:tasks.rados.rados.0.trial039.stdout:784: expect (ObjNum 350 snap 94 seq_num 350) 2026-02-25T08:45:02.148 INFO:tasks.rados.rados.0.trial039.stdout:785: read oid 42 snap 94 2026-02-25T08:45:02.148 INFO:tasks.rados.rados.0.trial039.stdout:785: expect (ObjNum 256 snap 71 seq_num 256) 2026-02-25T08:45:02.148 INFO:tasks.rados.rados.0.trial039.stdout:786: snap_create 2026-02-25T08:45:02.183 INFO:tasks.rados.rados.0.trial039.stdout:785: expect (ObjNum 256 snap 71 seq_num 256) 2026-02-25T08:45:02.183 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:02.203 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:02.232 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2275222 2026-02-25T08:45:02.232 INFO:tasks.rados.rados.0.trial039.stdout:781: finishing write tid 1 to trial03942481-50 2026-02-25T08:45:02.232 INFO:tasks.rados.rados.0.trial039.stdout:782: expect (ObjNum 301 snap 81 seq_num 301) 2026-02-25T08:45:02.237 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 868425 2026-02-25T08:45:02.262 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 504912 2026-02-25T08:45:02.331 INFO:tasks.rados.rados.0.trial039.stdout:784: expect (ObjNum 350 snap 94 seq_num 350) 2026-02-25T08:45:02.335 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1432822 2026-02-25T08:45:02.442 INFO:tasks.rados.rados.0.trial039.stdout:781: finishing write tid 2 to trial03942481-50 2026-02-25T08:45:02.442 INFO:tasks.rados.rados.0.trial039.stdout:781: finishing write tid 3 to trial03942481-50 2026-02-25T08:45:02.442 INFO:tasks.rados.rados.0.trial039.stdout:781: finishing write tid 4 to trial03942481-50 2026-02-25T08:45:02.443 INFO:tasks.rados.rados.0.trial039.stdout:781: oid 50 updating version 0 to 165 2026-02-25T08:45:02.443 INFO:tasks.rados.rados.0.trial039.stdout:781: oid 50 updating version 165 to 167 2026-02-25T08:45:02.443 INFO:tasks.rados.rados.0.trial039.stdout:781: oid 50 version 167 is already newer than 167 2026-02-25T08:45:02.443 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 50 v 167 (ObjNum 381 snap 103 seq_num 381) dirty exists 2026-02-25T08:45:02.443 INFO:tasks.rados.rados.0.trial039.stdout:781: left oid 50 (ObjNum 381 snap 103 seq_num 381) 2026-02-25T08:45:02.501 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:02.576 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:02.574+0000 7fcdd5b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:02.579 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:02.577+0000 7fcdd5b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:02.606 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:02.623 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:03.082 INFO:tasks.rados.rados.0.trial039.stdout:780: done (6 left) 2026-02-25T08:45:03.082 INFO:tasks.rados.rados.0.trial039.stdout:781: done (5 left) 2026-02-25T08:45:03.082 INFO:tasks.rados.rados.0.trial039.stdout:782: done (4 left) 2026-02-25T08:45:03.082 INFO:tasks.rados.rados.0.trial039.stdout:783: done (3 left) 2026-02-25T08:45:03.082 INFO:tasks.rados.rados.0.trial039.stdout:784: done (2 left) 2026-02-25T08:45:03.082 INFO:tasks.rados.rados.0.trial039.stdout:785: done (1 left) 2026-02-25T08:45:03.082 INFO:tasks.rados.rados.0.trial039.stdout:786: done (0 left) 2026-02-25T08:45:03.082 INFO:tasks.rados.rados.0.trial039.stdout:787: delete oid 17 current snap is 104 2026-02-25T08:45:03.097 INFO:tasks.rados.rados.0.trial039.stdout:787: done (0 left) 2026-02-25T08:45:03.098 INFO:tasks.rados.rados.0.trial039.stdout:788: write oid 1 current snap is 104 2026-02-25T08:45:03.098 INFO:tasks.rados.rados.0.trial039.stdout:788: seq_num 383 ranges {437081=638017,1679074=582789} 2026-02-25T08:45:03.102 INFO:tasks.rados.rados.0.trial039.stdout:788: writing trial03942481-1 from 437081 to 1075098 tid 1 2026-02-25T08:45:03.107 INFO:tasks.rados.rados.0.trial039.stdout:788: writing trial03942481-1 from 1679074 to 2261863 tid 2 2026-02-25T08:45:03.107 INFO:tasks.rados.rados.0.trial039.stdout:789: snap_remove snap 103 2026-02-25T08:45:04.082 INFO:tasks.rados.rados.0.trial039.stdout:789: done (1 left) 2026-02-25T08:45:04.083 INFO:tasks.rados.rados.0.trial039.stdout:790: read oid 7 snap 89 2026-02-25T08:45:04.083 INFO:tasks.rados.rados.0.trial039.stdout:790: expect deleted 2026-02-25T08:45:04.083 INFO:tasks.rados.rados.0.trial039.stdout:791: read oid 4 snap 87 2026-02-25T08:45:04.083 INFO:tasks.rados.rados.0.trial039.stdout:791: expect deleted 2026-02-25T08:45:04.083 INFO:tasks.rados.rados.0.trial039.stdout:792: snap_remove snap 101 2026-02-25T08:45:04.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~209s 2026-02-25T08:45:04.624 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:04.700 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:04.698+0000 7fc1b0f7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:04.703 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:04.702+0000 7fc1b0f7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:04.731 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:04.748 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:05.084 INFO:tasks.rados.rados.0.trial039.stdout:792: done (3 left) 2026-02-25T08:45:05.084 INFO:tasks.rados.rados.0.trial039.stdout:793: delete oid 36 current snap is 104 2026-02-25T08:45:05.084 INFO:tasks.rados.rados.0.trial039.stdout:788: finishing write tid 1 to trial03942481-1 2026-02-25T08:45:05.084 INFO:tasks.rados.rados.0.trial039.stdout:788: finishing write tid 2 to trial03942481-1 2026-02-25T08:45:05.084 INFO:tasks.rados.rados.0.trial039.stdout:788: finishing write tid 3 to trial03942481-1 2026-02-25T08:45:05.084 INFO:tasks.rados.rados.0.trial039.stdout:788: finishing write tid 4 to trial03942481-1 2026-02-25T08:45:05.084 INFO:tasks.rados.rados.0.trial039.stdout:788: oid 1 updating version 0 to 74 2026-02-25T08:45:05.085 INFO:tasks.rados.rados.0.trial039.stdout:788: oid 1 version 74 is already newer than 74 2026-02-25T08:45:05.085 INFO:tasks.rados.rados.0.trial039.stdout:788: oid 1 version 74 is already newer than 74 2026-02-25T08:45:05.085 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 1 v 74 (ObjNum 383 snap 104 seq_num 383) dirty exists 2026-02-25T08:45:05.085 INFO:tasks.rados.rados.0.trial039.stdout:788: left oid 1 (ObjNum 383 snap 104 seq_num 383) 2026-02-25T08:45:05.127 INFO:tasks.rados.rados.0.trial039.stdout:788: done (3 left) 2026-02-25T08:45:05.127 INFO:tasks.rados.rados.0.trial039.stdout:790: done (2 left) 2026-02-25T08:45:05.127 INFO:tasks.rados.rados.0.trial039.stdout:791: done (1 left) 2026-02-25T08:45:05.127 INFO:tasks.rados.rados.0.trial039.stdout:793: done (0 left) 2026-02-25T08:45:05.127 INFO:tasks.rados.rados.0.trial039.stdout:794: read oid 29 snap -1 2026-02-25T08:45:05.127 INFO:tasks.rados.rados.0.trial039.stdout:794: expect (ObjNum 324 snap 84 seq_num 324) 2026-02-25T08:45:05.127 INFO:tasks.rados.rados.0.trial039.stdout:795: snap_remove snap 87 2026-02-25T08:45:06.084 INFO:tasks.rados.rados.0.trial039.stdout:795: done (1 left) 2026-02-25T08:45:06.084 INFO:tasks.rados.rados.0.trial039.stdout:796: read oid 40 snap -1 2026-02-25T08:45:06.085 INFO:tasks.rados.rados.0.trial039.stdout:796: expect (ObjNum 378 snap 101 seq_num 378) 2026-02-25T08:45:06.085 INFO:tasks.rados.rados.0.trial039.stdout:797: delete oid 17 current snap is 104 2026-02-25T08:45:06.085 INFO:tasks.rados.rados.0.trial039.stdout:794: expect (ObjNum 324 snap 84 seq_num 324) 2026-02-25T08:45:06.228 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1542513 2026-02-25T08:45:06.270 INFO:tasks.rados.rados.0.trial039.stdout:794: done (2 left) 2026-02-25T08:45:06.270 INFO:tasks.rados.rados.0.trial039.stdout:797: done (1 left) 2026-02-25T08:45:06.270 INFO:tasks.rados.rados.0.trial039.stdout:798: read oid 44 snap -1 2026-02-25T08:45:06.271 INFO:tasks.rados.rados.0.trial039.stdout:798: expect deleted 2026-02-25T08:45:06.271 INFO:tasks.rados.rados.0.trial039.stdout:799: write (excl) oid 1 current snap is 104 2026-02-25T08:45:06.271 INFO:tasks.rados.rados.0.trial039.stdout:796: expect (ObjNum 378 snap 101 seq_num 378) 2026-02-25T08:45:06.283 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2125992 2026-02-25T08:45:06.343 INFO:tasks.rados.rados.0.trial039.stdout:799: seq_num 386 ranges {487502=463764,1738167=446469} 2026-02-25T08:45:06.347 INFO:tasks.rados.rados.0.trial039.stdout:799: writing trial03942481-1 from 487502 to 951266 tid 1 2026-02-25T08:45:06.352 INFO:tasks.rados.rados.0.trial039.stdout:799: writing trial03942481-1 from 1738167 to 2184636 tid 2 2026-02-25T08:45:06.352 INFO:tasks.rados.rados.0.trial039.stdout:796: done (2 left) 2026-02-25T08:45:06.352 INFO:tasks.rados.rados.0.trial039.stdout:798: done (1 left) 2026-02-25T08:45:06.352 INFO:tasks.rados.rados.0.trial039.stdout:800: snap_create 2026-02-25T08:45:06.363 INFO:tasks.rados.rados.0.trial039.stdout:799: finishing write tid 1 to trial03942481-1 2026-02-25T08:45:06.375 INFO:tasks.rados.rados.0.trial039.stdout:799: finishing write tid 2 to trial03942481-1 2026-02-25T08:45:06.380 INFO:tasks.rados.rados.0.trial039.stdout:799: finishing write tid 3 to trial03942481-1 2026-02-25T08:45:06.382 INFO:tasks.rados.rados.0.trial039.stdout:799: finishing write tid 4 to trial03942481-1 2026-02-25T08:45:06.382 INFO:tasks.rados.rados.0.trial039.stdout:799: oid 1 updating version 0 to 77 2026-02-25T08:45:06.382 INFO:tasks.rados.rados.0.trial039.stdout:799: oid 1 version 77 is already newer than 77 2026-02-25T08:45:06.383 INFO:tasks.rados.rados.0.trial039.stdout:799: oid 1 version 77 is already newer than 75 2026-02-25T08:45:06.383 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 1 v 77 (ObjNum 386 snap 104 seq_num 386) dirty exists 2026-02-25T08:45:06.383 INFO:tasks.rados.rados.0.trial039.stdout:799: left oid 1 (ObjNum 386 snap 104 seq_num 386) 2026-02-25T08:45:06.748 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:06.826 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:06.823+0000 7f1ea21dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:06.828 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:06.827+0000 7f1ea21dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:06.855 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:06.872 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:06.977 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:07.050 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:07.053 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:07.154 INFO:tasks.rados.rados.0.trial039.stdout:799: done (1 left) 2026-02-25T08:45:07.154 INFO:tasks.rados.rados.0.trial039.stdout:800: done (0 left) 2026-02-25T08:45:07.154 INFO:tasks.rados.rados.0.trial039.stdout:801: write oid 2 current snap is 105 2026-02-25T08:45:07.154 INFO:tasks.rados.rados.0.trial039.stdout:801: seq_num 387 ranges {504309=539013,1796649=590127} 2026-02-25T08:45:07.158 INFO:tasks.rados.rados.0.trial039.stdout:801: writing trial03942481-2 from 504309 to 1043322 tid 1 2026-02-25T08:45:07.163 INFO:tasks.rados.rados.0.trial039.stdout:801: writing trial03942481-2 from 1796649 to 2386776 tid 2 2026-02-25T08:45:07.163 INFO:tasks.rados.rados.0.trial039.stdout:802: read oid 6 snap -1 2026-02-25T08:45:07.164 INFO:tasks.rados.rados.0.trial039.stdout:802: expect (ObjNum 302 snap 81 seq_num 302) 2026-02-25T08:45:07.164 INFO:tasks.rados.rados.0.trial039.stdout:803: snap_remove snap 86 2026-02-25T08:45:08.154 INFO:tasks.rados.rados.0.trial039.stdout:803: done (2 left) 2026-02-25T08:45:08.154 INFO:tasks.rados.rados.0.trial039.stdout:804: snap_create 2026-02-25T08:45:08.155 INFO:tasks.rados.rados.0.trial039.stdout:802: expect (ObjNum 302 snap 81 seq_num 302) 2026-02-25T08:45:08.206 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:08.233 INFO:tasks.rados.rados.0.trial039.stdout:801: finishing write tid 1 to trial03942481-2 2026-02-25T08:45:08.233 INFO:tasks.rados.rados.0.trial039.stdout:801: finishing write tid 2 to trial03942481-2 2026-02-25T08:45:08.233 INFO:tasks.rados.rados.0.trial039.stdout:801: finishing write tid 3 to trial03942481-2 2026-02-25T08:45:08.233 INFO:tasks.rados.rados.0.trial039.stdout:801: finishing write tid 4 to trial03942481-2 2026-02-25T08:45:08.233 INFO:tasks.rados.rados.0.trial039.stdout:801: oid 2 updating version 0 to 155 2026-02-25T08:45:08.233 INFO:tasks.rados.rados.0.trial039.stdout:801: oid 2 version 155 is already newer than 155 2026-02-25T08:45:08.233 INFO:tasks.rados.rados.0.trial039.stdout:801: oid 2 version 155 is already newer than 155 2026-02-25T08:45:08.233 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 155 (ObjNum 387 snap 105 seq_num 387) dirty exists 2026-02-25T08:45:08.233 INFO:tasks.rados.rados.0.trial039.stdout:801: left oid 2 (ObjNum 387 snap 105 seq_num 387) 2026-02-25T08:45:08.873 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:08.951 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:08.948+0000 7fcb115cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:08.953 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:08.952+0000 7fcb115cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:08.980 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:08.998 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:09.161 INFO:tasks.rados.rados.0.trial039.stdout:801: done (2 left) 2026-02-25T08:45:09.161 INFO:tasks.rados.rados.0.trial039.stdout:802: done (1 left) 2026-02-25T08:45:09.161 INFO:tasks.rados.rados.0.trial039.stdout:804: done (0 left) 2026-02-25T08:45:09.161 INFO:tasks.rados.rados.0.trial039.stdout:805: write (excl) oid 40 current snap is 106 2026-02-25T08:45:09.161 INFO:tasks.rados.rados.0.trial039.stdout:805: seq_num 388 ranges {521116=730615,1971484=549632} 2026-02-25T08:45:09.166 INFO:tasks.rados.rados.0.trial039.stdout:805: writing trial03942481-40 from 521116 to 1251731 tid 1 2026-02-25T08:45:09.171 INFO:tasks.rados.rados.0.trial039.stdout:805: writing trial03942481-40 from 1971484 to 2521116 tid 2 2026-02-25T08:45:09.171 INFO:tasks.rados.rados.0.trial039.stdout:806: write (excl) oid 48 current snap is 106 2026-02-25T08:45:09.171 INFO:tasks.rados.rados.0.trial039.stdout:806: seq_num 389 ranges {537923=405864,1513613=593796} 2026-02-25T08:45:09.174 INFO:tasks.rados.rados.0.trial039.stdout:806: writing trial03942481-48 from 537923 to 943787 tid 1 2026-02-25T08:45:09.179 INFO:tasks.rados.rados.0.trial039.stdout:806: writing trial03942481-48 from 1513613 to 2107409 tid 2 2026-02-25T08:45:09.179 INFO:tasks.rados.rados.0.trial039.stdout:807: write (excl) oid 13 current snap is 106 2026-02-25T08:45:09.179 INFO:tasks.rados.rados.0.trial039.stdout:807: seq_num 390 ranges {554730=481113,1572095=737454} 2026-02-25T08:45:09.182 INFO:tasks.rados.rados.0.trial039.stdout:807: writing trial03942481-13 from 554730 to 1035843 tid 1 2026-02-25T08:45:09.187 INFO:tasks.rados.rados.0.trial039.stdout:807: writing trial03942481-13 from 1572095 to 2309549 tid 2 2026-02-25T08:45:09.188 INFO:tasks.rados.rados.0.trial039.stdout:808: write (excl) oid 43 current snap is 106 2026-02-25T08:45:09.188 INFO:tasks.rados.rados.0.trial039.stdout:808: seq_num 391 ranges {571537=556362,1630577=597465} 2026-02-25T08:45:09.192 INFO:tasks.rados.rados.0.trial039.stdout:808: writing trial03942481-43 from 571537 to 1127899 tid 1 2026-02-25T08:45:09.196 INFO:tasks.rados.rados.0.trial039.stdout:808: writing trial03942481-43 from 1630577 to 2228042 tid 2 2026-02-25T08:45:09.196 INFO:tasks.rados.rados.0.trial039.stdout:809: snap_remove snap 97 2026-02-25T08:45:09.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~215s 2026-02-25T08:45:10.165 INFO:tasks.rados.rados.0.trial039.stdout:809: done (4 left) 2026-02-25T08:45:10.166 INFO:tasks.rados.rados.0.trial039.stdout:810: write oid 3 current snap is 106 2026-02-25T08:45:10.166 INFO:tasks.rados.rados.0.trial039.stdout:805: finishing write tid 1 to trial03942481-40 2026-02-25T08:45:10.166 INFO:tasks.rados.rados.0.trial039.stdout:810: seq_num 392 ranges {588344=631611,1689059=741123} 2026-02-25T08:45:10.170 INFO:tasks.rados.rados.0.trial039.stdout:810: writing trial03942481-3 from 588344 to 1219955 tid 1 2026-02-25T08:45:10.177 INFO:tasks.rados.rados.0.trial039.stdout:810: writing trial03942481-3 from 1689059 to 2430182 tid 2 2026-02-25T08:45:10.177 INFO:tasks.rados.rados.0.trial039.stdout:811: snap_create 2026-02-25T08:45:10.177 INFO:tasks.rados.rados.0.trial039.stdout:807: finishing write tid 1 to trial03942481-13 2026-02-25T08:45:10.177 INFO:tasks.rados.rados.0.trial039.stdout:806: finishing write tid 1 to trial03942481-48 2026-02-25T08:45:10.177 INFO:tasks.rados.rados.0.trial039.stdout:805: finishing write tid 2 to trial03942481-40 2026-02-25T08:45:10.177 INFO:tasks.rados.rados.0.trial039.stdout:805: finishing write tid 3 to trial03942481-40 2026-02-25T08:45:10.177 INFO:tasks.rados.rados.0.trial039.stdout:805: finishing write tid 4 to trial03942481-40 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:805: oid 40 updating version 0 to 171 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:805: oid 40 version 171 is already newer than 171 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:805: oid 40 version 171 is already newer than 169 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 40 v 171 (ObjNum 388 snap 106 seq_num 388) dirty exists 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:805: left oid 40 (ObjNum 388 snap 106 seq_num 388) 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:807: finishing write tid 2 to trial03942481-13 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:807: finishing write tid 3 to trial03942481-13 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:807: finishing write tid 4 to trial03942481-13 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:807: oid 13 updating version 0 to 136 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:807: oid 13 updating version 136 to 138 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:807: oid 13 version 138 is already newer than 138 2026-02-25T08:45:10.178 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 13 v 138 (ObjNum 390 snap 106 seq_num 390) dirty exists 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:807: left oid 13 (ObjNum 390 snap 106 seq_num 390) 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:806: finishing write tid 2 to trial03942481-48 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:806: finishing write tid 3 to trial03942481-48 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:806: finishing write tid 4 to trial03942481-48 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:806: oid 48 updating version 0 to 46 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:806: oid 48 version 46 is already newer than 46 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:806: oid 48 version 46 is already newer than 44 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 48 v 46 (ObjNum 389 snap 106 seq_num 389) dirty exists 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:806: left oid 48 (ObjNum 389 snap 106 seq_num 389) 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:808: finishing write tid 1 to trial03942481-43 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:808: finishing write tid 2 to trial03942481-43 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:808: finishing write tid 3 to trial03942481-43 2026-02-25T08:45:10.179 INFO:tasks.rados.rados.0.trial039.stdout:808: finishing write tid 4 to trial03942481-43 2026-02-25T08:45:10.180 INFO:tasks.rados.rados.0.trial039.stdout:808: oid 43 updating version 0 to 125 2026-02-25T08:45:10.180 INFO:tasks.rados.rados.0.trial039.stdout:808: oid 43 version 125 is already newer than 125 2026-02-25T08:45:10.180 INFO:tasks.rados.rados.0.trial039.stdout:808: oid 43 version 125 is already newer than 123 2026-02-25T08:45:10.180 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 43 v 125 (ObjNum 391 snap 106 seq_num 391) dirty exists 2026-02-25T08:45:10.180 INFO:tasks.rados.rados.0.trial039.stdout:808: left oid 43 (ObjNum 391 snap 106 seq_num 391) 2026-02-25T08:45:10.210 INFO:tasks.rados.rados.0.trial039.stdout:810: finishing write tid 1 to trial03942481-3 2026-02-25T08:45:10.221 INFO:tasks.rados.rados.0.trial039.stdout:810: finishing write tid 2 to trial03942481-3 2026-02-25T08:45:10.229 INFO:tasks.rados.rados.0.trial039.stdout:810: finishing write tid 3 to trial03942481-3 2026-02-25T08:45:10.230 INFO:tasks.rados.rados.0.trial039.stdout:810: finishing write tid 4 to trial03942481-3 2026-02-25T08:45:10.230 INFO:tasks.rados.rados.0.trial039.stdout:810: oid 3 updating version 0 to 281 2026-02-25T08:45:10.230 INFO:tasks.rados.rados.0.trial039.stdout:810: oid 3 version 281 is already newer than 281 2026-02-25T08:45:10.230 INFO:tasks.rados.rados.0.trial039.stdout:810: oid 3 version 281 is already newer than 281 2026-02-25T08:45:10.230 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 3 v 281 (ObjNum 392 snap 106 seq_num 392) dirty exists 2026-02-25T08:45:10.230 INFO:tasks.rados.rados.0.trial039.stdout:810: left oid 3 (ObjNum 392 snap 106 seq_num 392) 2026-02-25T08:45:10.999 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:11.076 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:11.074+0000 7f397fa5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:11.079 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:11.077+0000 7f397fa5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:11.106 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:11.123 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:11.167 INFO:tasks.rados.rados.0.trial039.stdout:805: done (5 left) 2026-02-25T08:45:11.167 INFO:tasks.rados.rados.0.trial039.stdout:806: done (4 left) 2026-02-25T08:45:11.167 INFO:tasks.rados.rados.0.trial039.stdout:807: done (3 left) 2026-02-25T08:45:11.167 INFO:tasks.rados.rados.0.trial039.stdout:808: done (2 left) 2026-02-25T08:45:11.168 INFO:tasks.rados.rados.0.trial039.stdout:810: done (1 left) 2026-02-25T08:45:11.168 INFO:tasks.rados.rados.0.trial039.stdout:811: done (0 left) 2026-02-25T08:45:11.168 INFO:tasks.rados.rados.0.trial039.stdout:812: snap_create 2026-02-25T08:45:12.055 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:12.127 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:12.130 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:12.131 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:12.170 INFO:tasks.rados.rados.0.trial039.stdout:812: done (0 left) 2026-02-25T08:45:12.170 INFO:tasks.rados.rados.0.trial039.stdout:813: read oid 17 snap -1 2026-02-25T08:45:12.170 INFO:tasks.rados.rados.0.trial039.stdout:813: expect deleted 2026-02-25T08:45:12.170 INFO:tasks.rados.rados.0.trial039.stdout:814: read oid 28 snap 98 2026-02-25T08:45:12.170 INFO:tasks.rados.rados.0.trial039.stdout:814: expect (ObjNum 355 snap 97 seq_num 355) 2026-02-25T08:45:12.170 INFO:tasks.rados.rados.0.trial039.stdout:815: write oid 46 current snap is 108 2026-02-25T08:45:12.170 INFO:tasks.rados.rados.0.trial039.stdout:815: seq_num 393 ranges {605151=706860,2031188=573963} 2026-02-25T08:45:12.174 INFO:tasks.rados.rados.0.trial039.stdout:815: writing trial03942481-46 from 605151 to 1312011 tid 1 2026-02-25T08:45:12.179 INFO:tasks.rados.rados.0.trial039.stdout:815: writing trial03942481-46 from 2031188 to 2605151 tid 2 2026-02-25T08:45:12.179 INFO:tasks.rados.rados.0.trial039.stdout:816: delete oid 4 current snap is 108 2026-02-25T08:45:12.225 INFO:tasks.rados.rados.0.trial039.stdout:813: done (3 left) 2026-02-25T08:45:12.225 INFO:tasks.rados.rados.0.trial039.stdout:816: done (2 left) 2026-02-25T08:45:12.225 INFO:tasks.rados.rados.0.trial039.stdout:817: snap_remove snap 94 2026-02-25T08:45:13.084 INFO:tasks.rados.rados.0.trial039.stdout:817: done (2 left) 2026-02-25T08:45:13.084 INFO:tasks.rados.rados.0.trial039.stdout:818: snap_create 2026-02-25T08:45:13.084 INFO:tasks.rados.rados.0.trial039.stdout:814: expect (ObjNum 355 snap 97 seq_num 355) 2026-02-25T08:45:13.124 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:13.155 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2434675 2026-02-25T08:45:13.199 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:13.197+0000 7f816d70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:13.201 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:13.200+0000 7f816d70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:13.214 INFO:tasks.rados.rados.0.trial039.stdout:815: finishing write tid 1 to trial03942481-46 2026-02-25T08:45:13.214 INFO:tasks.rados.rados.0.trial039.stdout:815: finishing write tid 2 to trial03942481-46 2026-02-25T08:45:13.214 INFO:tasks.rados.rados.0.trial039.stdout:815: finishing write tid 3 to trial03942481-46 2026-02-25T08:45:13.214 INFO:tasks.rados.rados.0.trial039.stdout:815: finishing write tid 4 to trial03942481-46 2026-02-25T08:45:13.214 INFO:tasks.rados.rados.0.trial039.stdout:815: oid 46 updating version 0 to 145 2026-02-25T08:45:13.214 INFO:tasks.rados.rados.0.trial039.stdout:815: oid 46 version 145 is already newer than 145 2026-02-25T08:45:13.214 INFO:tasks.rados.rados.0.trial039.stdout:815: oid 46 version 145 is already newer than 145 2026-02-25T08:45:13.214 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 46 v 145 (ObjNum 393 snap 108 seq_num 393) dirty exists 2026-02-25T08:45:13.214 INFO:tasks.rados.rados.0.trial039.stdout:815: left oid 46 (ObjNum 393 snap 108 seq_num 393) 2026-02-25T08:45:13.228 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:13.245 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:14.085 INFO:tasks.rados.rados.0.trial039.stdout:814: done (2 left) 2026-02-25T08:45:14.085 INFO:tasks.rados.rados.0.trial039.stdout:815: done (1 left) 2026-02-25T08:45:14.085 INFO:tasks.rados.rados.0.trial039.stdout:818: done (0 left) 2026-02-25T08:45:14.085 INFO:tasks.rados.rados.0.trial039.stdout:819: write oid 48 current snap is 109 2026-02-25T08:45:14.085 INFO:tasks.rados.rados.0.trial039.stdout:819: seq_num 395 ranges {638765=457358,1748152=604803} 2026-02-25T08:45:14.089 INFO:tasks.rados.rados.0.trial039.stdout:819: writing trial03942481-48 from 638765 to 1096123 tid 1 2026-02-25T08:45:14.094 INFO:tasks.rados.rados.0.trial039.stdout:819: writing trial03942481-48 from 1748152 to 2352955 tid 2 2026-02-25T08:45:14.094 INFO:tasks.rados.rados.0.trial039.stdout:820: snap_remove snap 104 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:820: done (1 left) 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:821: read oid 12 snap 76 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:821: expect (ObjNum 281 snap 75 seq_num 281) 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:822: snap_remove snap 107 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:819: finishing write tid 1 to trial03942481-48 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:819: finishing write tid 2 to trial03942481-48 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:819: finishing write tid 3 to trial03942481-48 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:819: finishing write tid 4 to trial03942481-48 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:819: oid 48 updating version 0 to 50 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:819: oid 48 version 50 is already newer than 50 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:819: oid 48 version 50 is already newer than 50 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 48 v 50 (ObjNum 395 snap 109 seq_num 395) dirty exists 2026-02-25T08:45:15.086 INFO:tasks.rados.rados.0.trial039.stdout:819: left oid 48 (ObjNum 395 snap 109 seq_num 395) 2026-02-25T08:45:15.246 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:15.321 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:15.319+0000 7f8dcb011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:15.324 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:15.323+0000 7f8dcb011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:15.351 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:15.368 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:15.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~220s 2026-02-25T08:45:16.087 INFO:tasks.rados.rados.0.trial039.stdout:819: done (2 left) 2026-02-25T08:45:16.087 INFO:tasks.rados.rados.0.trial039.stdout:822: done (1 left) 2026-02-25T08:45:16.087 INFO:tasks.rados.rados.0.trial039.stdout:823: write (excl) oid 24 current snap is 109 2026-02-25T08:45:16.087 INFO:tasks.rados.rados.0.trial039.stdout:823: seq_num 396 ranges {655572=648960,1922987=732585} 2026-02-25T08:45:16.092 INFO:tasks.rados.rados.0.trial039.stdout:823: writing trial03942481-24 from 655572 to 1304532 tid 1 2026-02-25T08:45:16.098 INFO:tasks.rados.rados.0.trial039.stdout:823: writing trial03942481-24 from 1922987 to 2655572 tid 2 2026-02-25T08:45:16.098 INFO:tasks.rados.rados.0.trial039.stdout:824: read oid 22 snap -1 2026-02-25T08:45:16.098 INFO:tasks.rados.rados.0.trial039.stdout:824: expect (ObjNum 295 snap 78 seq_num 295) 2026-02-25T08:45:16.098 INFO:tasks.rados.rados.0.trial039.stdout:825: write oid 37 current snap is 109 2026-02-25T08:45:16.098 INFO:tasks.rados.rados.0.trial039.stdout:825: seq_num 397 ranges {672379=724209,1865116=608472} 2026-02-25T08:45:16.103 INFO:tasks.rados.rados.0.trial039.stdout:825: writing trial03942481-37 from 672379 to 1396588 tid 1 2026-02-25T08:45:16.107 INFO:tasks.rados.rados.0.trial039.stdout:825: writing trial03942481-37 from 1865116 to 2473588 tid 2 2026-02-25T08:45:16.107 INFO:tasks.rados.rados.0.trial039.stdout:826: snap_create 2026-02-25T08:45:16.107 INFO:tasks.rados.rados.0.trial039.stdout:821: expect (ObjNum 281 snap 75 seq_num 281) 2026-02-25T08:45:16.220 INFO:tasks.rados.rados.0.trial039.stdout:824: expect (ObjNum 295 snap 78 seq_num 295) 2026-02-25T08:45:16.272 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 162124 2026-02-25T08:45:16.305 INFO:tasks.rados.rados.0.trial039.stdout:823: finishing write tid 1 to trial03942481-24 2026-02-25T08:45:16.305 INFO:tasks.rados.rados.0.trial039.stdout:823: finishing write tid 2 to trial03942481-24 2026-02-25T08:45:16.305 INFO:tasks.rados.rados.0.trial039.stdout:823: finishing write tid 3 to trial03942481-24 2026-02-25T08:45:16.305 INFO:tasks.rados.rados.0.trial039.stdout:825: finishing write tid 1 to trial03942481-37 2026-02-25T08:45:16.305 INFO:tasks.rados.rados.0.trial039.stdout:823: finishing write tid 4 to trial03942481-24 2026-02-25T08:45:16.305 INFO:tasks.rados.rados.0.trial039.stdout:823: oid 24 updating version 0 to 69 2026-02-25T08:45:16.305 INFO:tasks.rados.rados.0.trial039.stdout:823: oid 24 version 69 is already newer than 69 2026-02-25T08:45:16.305 INFO:tasks.rados.rados.0.trial039.stdout:823: oid 24 version 69 is already newer than 67 2026-02-25T08:45:16.305 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 24 v 69 (ObjNum 396 snap 109 seq_num 396) dirty exists 2026-02-25T08:45:16.305 INFO:tasks.rados.rados.0.trial039.stdout:823: left oid 24 (ObjNum 396 snap 109 seq_num 396) 2026-02-25T08:45:16.306 INFO:tasks.rados.rados.0.trial039.stdout:825: finishing write tid 2 to trial03942481-37 2026-02-25T08:45:16.306 INFO:tasks.rados.rados.0.trial039.stdout:825: finishing write tid 3 to trial03942481-37 2026-02-25T08:45:16.306 INFO:tasks.rados.rados.0.trial039.stdout:825: finishing write tid 4 to trial03942481-37 2026-02-25T08:45:16.306 INFO:tasks.rados.rados.0.trial039.stdout:825: oid 37 updating version 0 to 159 2026-02-25T08:45:16.306 INFO:tasks.rados.rados.0.trial039.stdout:825: oid 37 version 159 is already newer than 159 2026-02-25T08:45:16.306 INFO:tasks.rados.rados.0.trial039.stdout:825: oid 37 version 159 is already newer than 159 2026-02-25T08:45:16.306 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 37 v 159 (ObjNum 397 snap 109 seq_num 397) dirty exists 2026-02-25T08:45:16.306 INFO:tasks.rados.rados.0.trial039.stdout:825: left oid 37 (ObjNum 397 snap 109 seq_num 397) 2026-02-25T08:45:17.093 INFO:tasks.rados.rados.0.trial039.stdout:821: done (4 left) 2026-02-25T08:45:17.093 INFO:tasks.rados.rados.0.trial039.stdout:823: done (3 left) 2026-02-25T08:45:17.093 INFO:tasks.rados.rados.0.trial039.stdout:824: done (2 left) 2026-02-25T08:45:17.094 INFO:tasks.rados.rados.0.trial039.stdout:825: done (1 left) 2026-02-25T08:45:17.094 INFO:tasks.rados.rados.0.trial039.stdout:826: done (0 left) 2026-02-25T08:45:17.094 INFO:tasks.rados.rados.0.trial039.stdout:827: write oid 34 current snap is 110 2026-02-25T08:45:17.094 INFO:tasks.rados.rados.0.trial039.stdout:827: seq_num 398 ranges {689186=799458,1923598=752130} 2026-02-25T08:45:17.099 INFO:tasks.rados.rados.0.trial039.stdout:827: writing trial03942481-34 from 689186 to 1488644 tid 1 2026-02-25T08:45:17.104 INFO:tasks.rados.rados.0.trial039.stdout:827: writing trial03942481-34 from 1923598 to 2675728 tid 2 2026-02-25T08:45:17.104 INFO:tasks.rados.rados.0.trial039.stdout:828: snap_create 2026-02-25T08:45:17.132 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:17.147 INFO:tasks.rados.rados.0.trial039.stdout:827: finishing write tid 1 to trial03942481-34 2026-02-25T08:45:17.154 INFO:tasks.rados.rados.0.trial039.stdout:827: finishing write tid 2 to trial03942481-34 2026-02-25T08:45:17.160 INFO:tasks.rados.rados.0.trial039.stdout:827: finishing write tid 3 to trial03942481-34 2026-02-25T08:45:17.161 INFO:tasks.rados.rados.0.trial039.stdout:827: finishing write tid 4 to trial03942481-34 2026-02-25T08:45:17.161 INFO:tasks.rados.rados.0.trial039.stdout:827: oid 34 updating version 0 to 175 2026-02-25T08:45:17.162 INFO:tasks.rados.rados.0.trial039.stdout:827: oid 34 version 175 is already newer than 175 2026-02-25T08:45:17.162 INFO:tasks.rados.rados.0.trial039.stdout:827: oid 34 version 175 is already newer than 175 2026-02-25T08:45:17.162 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 175 (ObjNum 398 snap 110 seq_num 398) dirty exists 2026-02-25T08:45:17.162 INFO:tasks.rados.rados.0.trial039.stdout:827: left oid 34 (ObjNum 398 snap 110 seq_num 398) 2026-02-25T08:45:17.204 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:17.208 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:17.369 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:17.448 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:17.445+0000 7f270145a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:17.450 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:17.449+0000 7f270145a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:17.477 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:17.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:18.096 INFO:tasks.rados.rados.0.trial039.stdout:827: done (1 left) 2026-02-25T08:45:18.096 INFO:tasks.rados.rados.0.trial039.stdout:828: done (0 left) 2026-02-25T08:45:18.096 INFO:tasks.rados.rados.0.trial039.stdout:829: snap_remove snap 110 2026-02-25T08:45:19.098 INFO:tasks.rados.rados.0.trial039.stdout:829: done (0 left) 2026-02-25T08:45:19.099 INFO:tasks.rados.rados.0.trial039.stdout:830: read oid 44 snap -1 2026-02-25T08:45:19.099 INFO:tasks.rados.rados.0.trial039.stdout:830: expect deleted 2026-02-25T08:45:19.099 INFO:tasks.rados.rados.0.trial039.stdout:831: snap_remove snap 102 2026-02-25T08:45:19.496 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:19.574 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:19.571+0000 7fb25f2b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:19.576 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:19.575+0000 7fb25f2b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:19.603 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:19.620 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:20.104 INFO:tasks.rados.rados.0.trial039.stdout:831: done (1 left) 2026-02-25T08:45:20.104 INFO:tasks.rados.rados.0.trial039.stdout:832: write oid 5 current snap is 111 2026-02-25T08:45:20.105 INFO:tasks.rados.rados.0.trial039.stdout:832: seq_num 399 ranges {705993=474707,1582080=612141} 2026-02-25T08:45:20.109 INFO:tasks.rados.rados.0.trial039.stdout:832: writing trial03942481-5 from 705993 to 1180700 tid 1 2026-02-25T08:45:20.114 INFO:tasks.rados.rados.0.trial039.stdout:832: writing trial03942481-5 from 1582080 to 2194221 tid 2 2026-02-25T08:45:20.114 INFO:tasks.rados.rados.0.trial039.stdout:833: read oid 26 snap -1 2026-02-25T08:45:20.115 INFO:tasks.rados.rados.0.trial039.stdout:833: expect (ObjNum 358 snap 97 seq_num 358) 2026-02-25T08:45:20.115 INFO:tasks.rados.rados.0.trial039.stdout:834: write (excl) oid 4 current snap is 111 2026-02-25T08:45:20.115 INFO:tasks.rados.rados.0.trial039.stdout:834: seq_num 400 ranges {722800=549956,2040562=682238} 2026-02-25T08:45:20.119 INFO:tasks.rados.rados.0.trial039.stdout:834: writing trial03942481-4 from 722800 to 1272756 tid 1 2026-02-25T08:45:20.125 INFO:tasks.rados.rados.0.trial039.stdout:834: writing trial03942481-4 from 2040562 to 2722800 tid 2 2026-02-25T08:45:20.126 INFO:tasks.rados.rados.0.trial039.stdout:835: write (excl) oid 33 current snap is 111 2026-02-25T08:45:20.126 INFO:tasks.rados.rados.0.trial039.stdout:835: seq_num 401 ranges {739607=625205,1982691=499457} 2026-02-25T08:45:20.131 INFO:tasks.rados.rados.0.trial039.stdout:835: writing trial03942481-33 from 739607 to 1364812 tid 1 2026-02-25T08:45:20.136 INFO:tasks.rados.rados.0.trial039.stdout:835: writing trial03942481-33 from 1982691 to 2482148 tid 2 2026-02-25T08:45:20.136 INFO:tasks.rados.rados.0.trial039.stdout:830: done (4 left) 2026-02-25T08:45:20.136 INFO:tasks.rados.rados.0.trial039.stdout:836: read oid 6 snap -1 2026-02-25T08:45:20.136 INFO:tasks.rados.rados.0.trial039.stdout:836: expect (ObjNum 302 snap 81 seq_num 302) 2026-02-25T08:45:20.137 INFO:tasks.rados.rados.0.trial039.stdout:837: delete oid 7 current snap is 111 2026-02-25T08:45:20.141 INFO:tasks.rados.rados.0.trial039.stdout:833: expect (ObjNum 358 snap 97 seq_num 358) 2026-02-25T08:45:20.166 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1776993 2026-02-25T08:45:20.169 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 896310 2026-02-25T08:45:20.181 INFO:tasks.rados.rados.0.trial039.stdout:833: done (5 left) 2026-02-25T08:45:20.181 INFO:tasks.rados.rados.0.trial039.stdout:837: done (4 left) 2026-02-25T08:45:20.181 INFO:tasks.rados.rados.0.trial039.stdout:838: write (excl) oid 26 current snap is 111 2026-02-25T08:45:20.181 INFO:tasks.rados.rados.0.trial039.stdout:834: finishing write tid 1 to trial03942481-4 2026-02-25T08:45:20.181 INFO:tasks.rados.rados.0.trial039.stdout:836: expect (ObjNum 302 snap 81 seq_num 302) 2026-02-25T08:45:20.259 INFO:tasks.rados.rados.0.trial039.stdout:834: finishing write tid 2 to trial03942481-4 2026-02-25T08:45:20.260 INFO:tasks.rados.rados.0.trial039.stdout:834: finishing write tid 3 to trial03942481-4 2026-02-25T08:45:20.260 INFO:tasks.rados.rados.0.trial039.stdout:838: seq_num 403 ranges {773221=492056,1816008=503126} 2026-02-25T08:45:20.265 INFO:tasks.rados.rados.0.trial039.stdout:838: writing trial03942481-26 from 773221 to 1265277 tid 1 2026-02-25T08:45:20.269 INFO:tasks.rados.rados.0.trial039.stdout:838: writing trial03942481-26 from 1816008 to 2319134 tid 2 2026-02-25T08:45:20.269 INFO:tasks.rados.rados.0.trial039.stdout:836: done (4 left) 2026-02-25T08:45:20.269 INFO:tasks.rados.rados.0.trial039.stdout:839: write oid 30 current snap is 111 2026-02-25T08:45:20.270 INFO:tasks.rados.rados.0.trial039.stdout:834: finishing write tid 4 to trial03942481-4 2026-02-25T08:45:20.270 INFO:tasks.rados.rados.0.trial039.stdout:834: oid 4 updating version 0 to 143 2026-02-25T08:45:20.270 INFO:tasks.rados.rados.0.trial039.stdout:834: oid 4 version 143 is already newer than 143 2026-02-25T08:45:20.270 INFO:tasks.rados.rados.0.trial039.stdout:834: oid 4 version 143 is already newer than 141 2026-02-25T08:45:20.270 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 143 (ObjNum 400 snap 111 seq_num 400) dirty exists 2026-02-25T08:45:20.270 INFO:tasks.rados.rados.0.trial039.stdout:834: left oid 4 (ObjNum 400 snap 111 seq_num 400) 2026-02-25T08:45:20.270 INFO:tasks.rados.rados.0.trial039.stdout:839: seq_num 404 ranges {790028=567305,1874490=763137} 2026-02-25T08:45:20.274 INFO:tasks.rados.rados.0.trial039.stdout:839: writing trial03942481-30 from 790028 to 1357333 tid 1 2026-02-25T08:45:20.279 INFO:tasks.rados.rados.0.trial039.stdout:839: writing trial03942481-30 from 1874490 to 2637627 tid 2 2026-02-25T08:45:20.279 INFO:tasks.rados.rados.0.trial039.stdout:834: done (4 left) 2026-02-25T08:45:20.279 INFO:tasks.rados.rados.0.trial039.stdout:840: write (excl) oid 19 current snap is 111 2026-02-25T08:45:20.280 INFO:tasks.rados.rados.0.trial039.stdout:832: finishing write tid 1 to trial03942481-5 2026-02-25T08:45:20.280 INFO:tasks.rados.rados.0.trial039.stdout:840: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2026-02-25T08:45:20.283 INFO:tasks.rados.rados.0.trial039.stdout:840: writing trial03942481-19 from 406835 to 1049389 tid 1 2026-02-25T08:45:20.287 INFO:tasks.rados.rados.0.trial039.stdout:840: writing trial03942481-19 from 1532972 to 2039767 tid 2 2026-02-25T08:45:20.290 INFO:tasks.rados.rados.0.trial039.stdout:840: writing trial03942481-19 from 2772312 to 2806835 tid 3 2026-02-25T08:45:20.290 INFO:tasks.rados.rados.0.trial039.stdout:841: write (excl) oid 47 current snap is 111 2026-02-25T08:45:20.290 INFO:tasks.rados.rados.0.trial039.stdout:835: finishing write tid 1 to trial03942481-33 2026-02-25T08:45:20.291 INFO:tasks.rados.rados.0.trial039.stdout:841: seq_num 406 ranges {423642=717803,1875101=766806} 2026-02-25T08:45:20.294 INFO:tasks.rados.rados.0.trial039.stdout:841: writing trial03942481-47 from 423642 to 1141445 tid 1 2026-02-25T08:45:20.300 INFO:tasks.rados.rados.0.trial039.stdout:841: writing trial03942481-47 from 1875101 to 2641907 tid 2 2026-02-25T08:45:20.300 INFO:tasks.rados.rados.0.trial039.stdout:842: write oid 2 current snap is 111 2026-02-25T08:45:20.300 INFO:tasks.rados.rados.0.trial039.stdout:842: seq_num 407 ranges {440449=793052,1933583=510464} 2026-02-25T08:45:20.305 INFO:tasks.rados.rados.0.trial039.stdout:842: writing trial03942481-2 from 440449 to 1233501 tid 1 2026-02-25T08:45:20.310 INFO:tasks.rados.rados.0.trial039.stdout:842: writing trial03942481-2 from 1933583 to 2444047 tid 2 2026-02-25T08:45:20.310 INFO:tasks.rados.rados.0.trial039.stdout:843: write oid 38 current snap is 111 2026-02-25T08:45:20.310 INFO:tasks.rados.rados.0.trial039.stdout:832: finishing write tid 2 to trial03942481-5 2026-02-25T08:45:20.310 INFO:tasks.rados.rados.0.trial039.stdout:843: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2026-02-25T08:45:20.313 INFO:tasks.rados.rados.0.trial039.stdout:843: writing trial03942481-38 from 457256 to 925557 tid 1 2026-02-25T08:45:20.319 INFO:tasks.rados.rados.0.trial039.stdout:843: writing trial03942481-38 from 1592065 to 2362540 tid 2 2026-02-25T08:45:20.322 INFO:tasks.rados.rados.0.trial039.stdout:843: writing trial03942481-38 from 2854581 to 2857256 tid 3 2026-02-25T08:45:20.322 INFO:tasks.rados.rados.0.trial039.stdout:844: read oid 14 snap -1 2026-02-25T08:45:20.322 INFO:tasks.rados.rados.0.trial039.stdout:844: expect (ObjNum 365 snap 97 seq_num 365) 2026-02-25T08:45:20.322 INFO:tasks.rados.rados.0.trial039.stdout:845: delete oid 46 current snap is 111 2026-02-25T08:45:20.322 INFO:tasks.rados.rados.0.trial039.stdout:832: finishing write tid 3 to trial03942481-5 2026-02-25T08:45:20.322 INFO:tasks.rados.rados.0.trial039.stdout:832: finishing write tid 4 to trial03942481-5 2026-02-25T08:45:20.322 INFO:tasks.rados.rados.0.trial039.stdout:832: oid 5 updating version 0 to 151 2026-02-25T08:45:20.322 INFO:tasks.rados.rados.0.trial039.stdout:832: oid 5 version 151 is already newer than 151 2026-02-25T08:45:20.322 INFO:tasks.rados.rados.0.trial039.stdout:832: oid 5 version 151 is already newer than 151 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 5 v 151 (ObjNum 399 snap 111 seq_num 399) dirty exists 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:832: left oid 5 (ObjNum 399 snap 111 seq_num 399) 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:835: finishing write tid 2 to trial03942481-33 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:835: finishing write tid 3 to trial03942481-33 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:835: finishing write tid 4 to trial03942481-33 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:835: oid 33 updating version 0 to 126 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:835: oid 33 updating version 126 to 128 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:835: oid 33 version 128 is already newer than 128 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 33 v 128 (ObjNum 401 snap 111 seq_num 401) dirty exists 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:835: left oid 33 (ObjNum 401 snap 111 seq_num 401) 2026-02-25T08:45:20.323 INFO:tasks.rados.rados.0.trial039.stdout:838: finishing write tid 1 to trial03942481-26 2026-02-25T08:45:20.343 INFO:tasks.rados.rados.0.trial039.stdout:841: finishing write tid 1 to trial03942481-47 2026-02-25T08:45:20.348 INFO:tasks.rados.rados.0.trial039.stdout:843: finishing write tid 1 to trial03942481-38 2026-02-25T08:45:20.354 INFO:tasks.rados.rados.0.trial039.stdout:838: finishing write tid 2 to trial03942481-26 2026-02-25T08:45:20.363 INFO:tasks.rados.rados.0.trial039.stdout:843: finishing write tid 2 to trial03942481-38 2026-02-25T08:45:20.367 INFO:tasks.rados.rados.0.trial039.stdout:838: finishing write tid 3 to trial03942481-26 2026-02-25T08:45:20.367 INFO:tasks.rados.rados.0.trial039.stdout:843: finishing write tid 3 to trial03942481-38 2026-02-25T08:45:20.370 INFO:tasks.rados.rados.0.trial039.stdout:844: expect (ObjNum 365 snap 97 seq_num 365) 2026-02-25T08:45:20.395 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1938486 2026-02-25T08:45:20.415 INFO:tasks.rados.rados.0.trial039.stdout:843: finishing write tid 4 to trial03942481-38 2026-02-25T08:45:20.416 INFO:tasks.rados.rados.0.trial039.stdout:841: finishing write tid 2 to trial03942481-47 2026-02-25T08:45:20.416 INFO:tasks.rados.rados.0.trial039.stdout:838: finishing write tid 4 to trial03942481-26 2026-02-25T08:45:20.416 INFO:tasks.rados.rados.0.trial039.stdout:838: oid 26 updating version 0 to 81 2026-02-25T08:45:20.416 INFO:tasks.rados.rados.0.trial039.stdout:838: oid 26 version 81 is already newer than 81 2026-02-25T08:45:20.416 INFO:tasks.rados.rados.0.trial039.stdout:838: oid 26 version 81 is already newer than 79 2026-02-25T08:45:20.416 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 26 v 81 (ObjNum 403 snap 111 seq_num 403) dirty exists 2026-02-25T08:45:20.416 INFO:tasks.rados.rados.0.trial039.stdout:838: left oid 26 (ObjNum 403 snap 111 seq_num 403) 2026-02-25T08:45:20.416 INFO:tasks.rados.rados.0.trial039.stdout:843: finishing write tid 5 to trial03942481-38 2026-02-25T08:45:20.416 INFO:tasks.rados.rados.0.trial039.stdout:843: oid 38 updating version 0 to 147 2026-02-25T08:45:20.416 INFO:tasks.rados.rados.0.trial039.stdout:843: oid 38 version 147 is already newer than 147 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:843: oid 38 version 147 is already newer than 147 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:843: oid 38 version 147 is already newer than 147 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 38 v 147 (ObjNum 408 snap 111 seq_num 408) dirty exists 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:843: left oid 38 (ObjNum 408 snap 111 seq_num 408) 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:839: finishing write tid 1 to trial03942481-30 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:832: done (9 left) 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:835: done (8 left) 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:838: done (7 left) 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:843: done (6 left) 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:844: done (5 left) 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:845: done (4 left) 2026-02-25T08:45:20.417 INFO:tasks.rados.rados.0.trial039.stdout:846: write (excl) oid 5 current snap is 111 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:841: finishing write tid 3 to trial03942481-47 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:839: finishing write tid 2 to trial03942481-30 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:841: finishing write tid 4 to trial03942481-47 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:841: oid 47 updating version 0 to 134 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:841: oid 47 version 134 is already newer than 134 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:841: oid 47 version 134 is already newer than 132 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 47 v 134 (ObjNum 406 snap 111 seq_num 406) dirty exists 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:841: left oid 47 (ObjNum 406 snap 111 seq_num 406) 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:839: finishing write tid 3 to trial03942481-30 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:839: finishing write tid 4 to trial03942481-30 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:839: oid 30 updating version 0 to 101 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:839: oid 30 version 101 is already newer than 101 2026-02-25T08:45:20.418 INFO:tasks.rados.rados.0.trial039.stdout:839: oid 30 version 101 is already newer than 101 2026-02-25T08:45:20.419 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 30 v 101 (ObjNum 404 snap 111 seq_num 404) dirty exists 2026-02-25T08:45:20.419 INFO:tasks.rados.rados.0.trial039.stdout:839: left oid 30 (ObjNum 404 snap 111 seq_num 404) 2026-02-25T08:45:20.419 INFO:tasks.rados.rados.0.trial039.stdout:846: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2026-02-25T08:45:20.419 INFO:tasks.rados.rados.0.trial039.stdout:846: writing trial03942481-5 from 490870 to 1109669 tid 1 2026-02-25T08:45:20.423 INFO:tasks.rados.rados.0.trial039.stdout:846: writing trial03942481-5 from 1592676 to 2366820 tid 2 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:846: writing trial03942481-5 from 2793074 to 2890870 tid 3 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:839: done (4 left) 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:841: done (3 left) 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:847: read oid 45 snap -1 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:847: expect deleted 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:848: read oid 47 snap -1 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:848: expect (ObjNum 406 snap 111 seq_num 406) 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:849: read oid 20 snap -1 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:849: expect (ObjNum 305 snap 81 seq_num 305) 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:850: read oid 42 snap -1 2026-02-25T08:45:20.427 INFO:tasks.rados.rados.0.trial039.stdout:850: expect (ObjNum 359 snap 97 seq_num 359) 2026-02-25T08:45:20.428 INFO:tasks.rados.rados.0.trial039.stdout:851: write (excl) oid 50 current snap is 111 2026-02-25T08:45:20.428 INFO:tasks.rados.rados.0.trial039.stdout:840: finishing write tid 1 to trial03942481-19 2026-02-25T08:45:20.428 INFO:tasks.rados.rados.0.trial039.stdout:851: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2026-02-25T08:45:20.429 INFO:tasks.rados.rados.0.trial039.stdout:851: writing trial03942481-50 from 507677 to 918078 tid 1 2026-02-25T08:45:20.433 INFO:tasks.rados.rados.0.trial039.stdout:851: writing trial03942481-50 from 1367511 to 1885313 tid 2 2026-02-25T08:45:20.437 INFO:tasks.rados.rados.0.trial039.stdout:851: writing trial03942481-50 from 2420497 to 2907677 tid 3 2026-02-25T08:45:20.437 INFO:tasks.rados.rados.0.trial039.stdout:852: snap_remove snap 105 2026-02-25T08:45:20.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~226s 2026-02-25T08:45:21.107 INFO:tasks.rados.rados.0.trial039.stdout:852: done (8 left) 2026-02-25T08:45:21.107 INFO:tasks.rados.rados.0.trial039.stdout:853: delete oid 12 current snap is 111 2026-02-25T08:45:21.107 INFO:tasks.rados.rados.0.trial039.stdout:840: finishing write tid 2 to trial03942481-19 2026-02-25T08:45:21.107 INFO:tasks.rados.rados.0.trial039.stdout:846: finishing write tid 1 to trial03942481-5 2026-02-25T08:45:21.107 INFO:tasks.rados.rados.0.trial039.stdout:848: expect (ObjNum 406 snap 111 seq_num 406) 2026-02-25T08:45:21.107 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 221007 2026-02-25T08:45:21.164 INFO:tasks.rados.rados.0.trial039.stdout:847: done (8 left) 2026-02-25T08:45:21.164 INFO:tasks.rados.rados.0.trial039.stdout:848: done (7 left) 2026-02-25T08:45:21.164 INFO:tasks.rados.rados.0.trial039.stdout:853: done (6 left) 2026-02-25T08:45:21.164 INFO:tasks.rados.rados.0.trial039.stdout:854: snap_create 2026-02-25T08:45:21.164 INFO:tasks.rados.rados.0.trial039.stdout:840: finishing write tid 3 to trial03942481-19 2026-02-25T08:45:21.164 INFO:tasks.rados.rados.0.trial039.stdout:850: expect (ObjNum 359 snap 97 seq_num 359) 2026-02-25T08:45:21.193 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1451667 2026-02-25T08:45:21.203 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 969827 2026-02-25T08:45:21.214 INFO:tasks.rados.rados.0.trial039.stdout:849: expect (ObjNum 305 snap 81 seq_num 305) 2026-02-25T08:45:21.335 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1013072 2026-02-25T08:45:21.385 INFO:tasks.rados.rados.0.trial039.stdout:846: finishing write tid 2 to trial03942481-5 2026-02-25T08:45:21.385 INFO:tasks.rados.rados.0.trial039.stdout:846: finishing write tid 3 to trial03942481-5 2026-02-25T08:45:21.385 INFO:tasks.rados.rados.0.trial039.stdout:840: finishing write tid 4 to trial03942481-19 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:840: finishing write tid 5 to trial03942481-19 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:840: oid 19 updating version 0 to 288 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:840: oid 19 version 288 is already newer than 288 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:840: oid 19 version 288 is already newer than 288 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:840: oid 19 version 288 is already newer than 285 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 19 v 288 (ObjNum 405 snap 111 seq_num 405) dirty exists 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:840: left oid 19 (ObjNum 405 snap 111 seq_num 405) 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:846: finishing write tid 4 to trial03942481-5 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:851: finishing write tid 1 to trial03942481-50 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:846: finishing write tid 5 to trial03942481-5 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:846: oid 5 updating version 0 to 157 2026-02-25T08:45:21.386 INFO:tasks.rados.rados.0.trial039.stdout:846: oid 5 version 157 is already newer than 154 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:846: oid 5 version 157 is already newer than 157 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:846: oid 5 version 157 is already newer than 157 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 5 v 157 (ObjNum 410 snap 111 seq_num 410) dirty exists 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:846: left oid 5 (ObjNum 410 snap 111 seq_num 410) 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:842: finishing write tid 1 to trial03942481-2 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:851: finishing write tid 2 to trial03942481-50 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:842: finishing write tid 2 to trial03942481-2 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:851: finishing write tid 3 to trial03942481-50 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:842: finishing write tid 3 to trial03942481-2 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:842: finishing write tid 4 to trial03942481-2 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:842: oid 2 updating version 0 to 165 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:842: oid 2 version 165 is already newer than 165 2026-02-25T08:45:21.387 INFO:tasks.rados.rados.0.trial039.stdout:842: oid 2 version 165 is already newer than 165 2026-02-25T08:45:21.388 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 165 (ObjNum 407 snap 111 seq_num 407) dirty exists 2026-02-25T08:45:21.388 INFO:tasks.rados.rados.0.trial039.stdout:842: left oid 2 (ObjNum 407 snap 111 seq_num 407) 2026-02-25T08:45:21.388 INFO:tasks.rados.rados.0.trial039.stdout:851: finishing write tid 4 to trial03942481-50 2026-02-25T08:45:21.388 INFO:tasks.rados.rados.0.trial039.stdout:851: finishing write tid 5 to trial03942481-50 2026-02-25T08:45:21.388 INFO:tasks.rados.rados.0.trial039.stdout:851: oid 50 updating version 0 to 180 2026-02-25T08:45:21.388 INFO:tasks.rados.rados.0.trial039.stdout:851: oid 50 version 180 is already newer than 180 2026-02-25T08:45:21.388 INFO:tasks.rados.rados.0.trial039.stdout:851: oid 50 version 180 is already newer than 177 2026-02-25T08:45:21.388 INFO:tasks.rados.rados.0.trial039.stdout:851: oid 50 version 180 is already newer than 180 2026-02-25T08:45:21.388 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 50 v 180 (ObjNum 411 snap 111 seq_num 411) dirty exists 2026-02-25T08:45:21.388 INFO:tasks.rados.rados.0.trial039.stdout:851: left oid 50 (ObjNum 411 snap 111 seq_num 411) 2026-02-25T08:45:21.621 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:21.698 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:21.696+0000 7fa454a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:21.701 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:21.700+0000 7fa454a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:21.728 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:21.745 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:22.112 INFO:tasks.rados.rados.0.trial039.stdout:840: done (6 left) 2026-02-25T08:45:22.112 INFO:tasks.rados.rados.0.trial039.stdout:842: done (5 left) 2026-02-25T08:45:22.112 INFO:tasks.rados.rados.0.trial039.stdout:846: done (4 left) 2026-02-25T08:45:22.112 INFO:tasks.rados.rados.0.trial039.stdout:849: done (3 left) 2026-02-25T08:45:22.113 INFO:tasks.rados.rados.0.trial039.stdout:850: done (2 left) 2026-02-25T08:45:22.113 INFO:tasks.rados.rados.0.trial039.stdout:851: done (1 left) 2026-02-25T08:45:22.113 INFO:tasks.rados.rados.0.trial039.stdout:854: done (0 left) 2026-02-25T08:45:22.113 INFO:tasks.rados.rados.0.trial039.stdout:855: write (excl) oid 9 current snap is 112 2026-02-25T08:45:22.113 INFO:tasks.rados.rados.0.trial039.stdout:855: seq_num 413 ranges {541291=560899,1884475=521471} 2026-02-25T08:45:22.116 INFO:tasks.rados.rados.0.trial039.stdout:855: writing trial03942481-9 from 541291 to 1102190 tid 1 2026-02-25T08:45:22.121 INFO:tasks.rados.rados.0.trial039.stdout:855: writing trial03942481-9 from 1884475 to 2405946 tid 2 2026-02-25T08:45:22.121 INFO:tasks.rados.rados.0.trial039.stdout:856: delete oid 39 current snap is 112 2026-02-25T08:45:22.155 INFO:tasks.rados.rados.0.trial039.stdout:856: done (1 left) 2026-02-25T08:45:22.155 INFO:tasks.rados.rados.0.trial039.stdout:857: read oid 19 snap -1 2026-02-25T08:45:22.155 INFO:tasks.rados.rados.0.trial039.stdout:857: expect (ObjNum 405 snap 111 seq_num 405) 2026-02-25T08:45:22.155 INFO:tasks.rados.rados.0.trial039.stdout:858: write (excl) oid 13 current snap is 112 2026-02-25T08:45:22.156 INFO:tasks.rados.rados.0.trial039.stdout:858: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2026-02-25T08:45:22.160 INFO:tasks.rados.rados.0.trial039.stdout:858: writing trial03942481-13 from 574905 to 1286302 tid 1 2026-02-25T08:45:22.164 INFO:tasks.rados.rados.0.trial039.stdout:858: writing trial03942481-13 from 1885086 to 2410226 tid 2 2026-02-25T08:45:22.167 INFO:tasks.rados.rados.0.trial039.stdout:858: writing trial03942481-13 from 2930189 to 2974905 tid 3 2026-02-25T08:45:22.167 INFO:tasks.rados.rados.0.trial039.stdout:859: snap_remove snap 76 2026-02-25T08:45:22.208 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:22.281 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:22.284 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:23.086 INFO:tasks.rados.rados.0.trial039.stdout:859: done (3 left) 2026-02-25T08:45:23.087 INFO:tasks.rados.rados.0.trial039.stdout:860: read oid 35 snap 99 2026-02-25T08:45:23.087 INFO:tasks.rados.rados.0.trial039.stdout:860: expect (ObjNum 356 snap 97 seq_num 356) 2026-02-25T08:45:23.087 INFO:tasks.rados.rados.0.trial039.stdout:861: delete oid 45 current snap is 112 2026-02-25T08:45:23.087 INFO:tasks.rados.rados.0.trial039.stdout:855: finishing write tid 1 to trial03942481-9 2026-02-25T08:45:23.087 INFO:tasks.rados.rados.0.trial039.stdout:857: expect (ObjNum 405 snap 111 seq_num 405) 2026-02-25T08:45:23.089 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 904717 2026-02-25T08:45:23.147 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:23.179 INFO:tasks.rados.rados.0.trial039.stdout:855: finishing write tid 2 to trial03942481-9 2026-02-25T08:45:23.179 INFO:tasks.rados.rados.0.trial039.stdout:855: finishing write tid 3 to trial03942481-9 2026-02-25T08:45:23.179 INFO:tasks.rados.rados.0.trial039.stdout:855: finishing write tid 4 to trial03942481-9 2026-02-25T08:45:23.179 INFO:tasks.rados.rados.0.trial039.stdout:855: oid 9 updating version 0 to 38 2026-02-25T08:45:23.179 INFO:tasks.rados.rados.0.trial039.stdout:855: oid 9 version 38 is already newer than 38 2026-02-25T08:45:23.179 INFO:tasks.rados.rados.0.trial039.stdout:855: oid 9 version 38 is already newer than 36 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 9 v 38 (ObjNum 413 snap 112 seq_num 413) dirty exists 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:855: left oid 9 (ObjNum 413 snap 112 seq_num 413) 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:858: finishing write tid 1 to trial03942481-13 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:858: finishing write tid 2 to trial03942481-13 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:858: finishing write tid 3 to trial03942481-13 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:858: finishing write tid 4 to trial03942481-13 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:858: finishing write tid 5 to trial03942481-13 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:858: oid 13 updating version 0 to 149 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:858: oid 13 updating version 149 to 152 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:858: oid 13 version 152 is already newer than 152 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:858: oid 13 version 152 is already newer than 152 2026-02-25T08:45:23.180 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 13 v 152 (ObjNum 415 snap 112 seq_num 415) dirty exists 2026-02-25T08:45:23.181 INFO:tasks.rados.rados.0.trial039.stdout:858: left oid 13 (ObjNum 415 snap 112 seq_num 415) 2026-02-25T08:45:23.181 INFO:tasks.rados.rados.0.trial039.stdout:855: done (4 left) 2026-02-25T08:45:23.181 INFO:tasks.rados.rados.0.trial039.stdout:857: done (3 left) 2026-02-25T08:45:23.181 INFO:tasks.rados.rados.0.trial039.stdout:858: done (2 left) 2026-02-25T08:45:23.181 INFO:tasks.rados.rados.0.trial039.stdout:861: done (1 left) 2026-02-25T08:45:23.181 INFO:tasks.rados.rados.0.trial039.stdout:862: snap_create 2026-02-25T08:45:23.181 INFO:tasks.rados.rados.0.trial039.stdout:860: expect (ObjNum 356 snap 97 seq_num 356) 2026-02-25T08:45:23.746 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:23.826 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:23.824+0000 7f24f4ffd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:23.828 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:23.827+0000 7f24f4ffd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:23.855 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:23.872 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:24.119 INFO:tasks.rados.rados.0.trial039.stdout:860: done (1 left) 2026-02-25T08:45:24.119 INFO:tasks.rados.rados.0.trial039.stdout:862: done (0 left) 2026-02-25T08:45:24.120 INFO:tasks.rados.rados.0.trial039.stdout:863: snap_remove snap 99 2026-02-25T08:45:25.124 INFO:tasks.rados.rados.0.trial039.stdout:863: done (0 left) 2026-02-25T08:45:25.125 INFO:tasks.rados.rados.0.trial039.stdout:864: write (excl) oid 32 current snap is 113 2026-02-25T08:45:25.125 INFO:tasks.rados.rados.0.trial039.stdout:864: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2026-02-25T08:45:25.128 INFO:tasks.rados.rados.0.trial039.stdout:864: writing trial03942481-32 from 608519 to 1070414 tid 1 2026-02-25T08:45:25.132 INFO:tasks.rados.rados.0.trial039.stdout:864: writing trial03942481-32 from 1602050 to 2130859 tid 2 2026-02-25T08:45:25.136 INFO:tasks.rados.rados.0.trial039.stdout:864: writing trial03942481-32 from 2585035 to 3008519 tid 3 2026-02-25T08:45:25.136 INFO:tasks.rados.rados.0.trial039.stdout:865: snap_remove snap 73 2026-02-25T08:45:25.874 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:25.951 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:25.949+0000 7f0b525b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:25.954 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:25.953+0000 7f0b525b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:25.982 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:25.998 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:25.998 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:26.128 INFO:tasks.rados.rados.0.trial039.stdout:865: done (1 left) 2026-02-25T08:45:26.128 INFO:tasks.rados.rados.0.trial039.stdout:866: snap_remove snap 109 2026-02-25T08:45:26.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~231s 2026-02-25T08:45:27.131 INFO:tasks.rados.rados.0.trial039.stdout:866: done (1 left) 2026-02-25T08:45:27.132 INFO:tasks.rados.rados.0.trial039.stdout:867: delete oid 37 current snap is 113 2026-02-25T08:45:27.132 INFO:tasks.rados.rados.0.trial039.stdout:864: finishing write tid 1 to trial03942481-32 2026-02-25T08:45:27.132 INFO:tasks.rados.rados.0.trial039.stdout:864: finishing write tid 2 to trial03942481-32 2026-02-25T08:45:27.132 INFO:tasks.rados.rados.0.trial039.stdout:864: finishing write tid 3 to trial03942481-32 2026-02-25T08:45:27.132 INFO:tasks.rados.rados.0.trial039.stdout:864: finishing write tid 4 to trial03942481-32 2026-02-25T08:45:27.132 INFO:tasks.rados.rados.0.trial039.stdout:864: finishing write tid 5 to trial03942481-32 2026-02-25T08:45:27.132 INFO:tasks.rados.rados.0.trial039.stdout:864: oid 32 updating version 0 to 103 2026-02-25T08:45:27.132 INFO:tasks.rados.rados.0.trial039.stdout:864: oid 32 updating version 103 to 106 2026-02-25T08:45:27.132 INFO:tasks.rados.rados.0.trial039.stdout:864: oid 32 version 106 is already newer than 106 2026-02-25T08:45:27.132 INFO:tasks.rados.rados.0.trial039.stdout:864: oid 32 version 106 is already newer than 106 2026-02-25T08:45:27.133 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 32 v 106 (ObjNum 417 snap 113 seq_num 417) dirty exists 2026-02-25T08:45:27.133 INFO:tasks.rados.rados.0.trial039.stdout:864: left oid 32 (ObjNum 417 snap 113 seq_num 417) 2026-02-25T08:45:27.177 INFO:tasks.rados.rados.0.trial039.stdout:864: done (1 left) 2026-02-25T08:45:27.178 INFO:tasks.rados.rados.0.trial039.stdout:867: done (0 left) 2026-02-25T08:45:27.178 INFO:tasks.rados.rados.0.trial039.stdout:868: write oid 25 current snap is 113 2026-02-25T08:45:27.178 INFO:tasks.rados.rados.0.trial039.stdout:868: seq_num 419 ranges {642133=728746,2119014=532478} 2026-02-25T08:45:27.183 INFO:tasks.rados.rados.0.trial039.stdout:868: writing trial03942481-25 from 642133 to 1370879 tid 1 2026-02-25T08:45:27.188 INFO:tasks.rados.rados.0.trial039.stdout:868: writing trial03942481-25 from 2119014 to 2651492 tid 2 2026-02-25T08:45:27.188 INFO:tasks.rados.rados.0.trial039.stdout:869: read oid 48 snap -1 2026-02-25T08:45:27.189 INFO:tasks.rados.rados.0.trial039.stdout:869: expect (ObjNum 395 snap 109 seq_num 395) 2026-02-25T08:45:27.189 INFO:tasks.rados.rados.0.trial039.stdout:870: snap_create 2026-02-25T08:45:27.208 INFO:tasks.rados.rados.0.trial039.stdout:869: expect (ObjNum 395 snap 109 seq_num 395) 2026-02-25T08:45:27.222 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1695817 2026-02-25T08:45:27.285 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:27.320 INFO:tasks.rados.rados.0.trial039.stdout:868: finishing write tid 1 to trial03942481-25 2026-02-25T08:45:27.320 INFO:tasks.rados.rados.0.trial039.stdout:868: finishing write tid 2 to trial03942481-25 2026-02-25T08:45:27.320 INFO:tasks.rados.rados.0.trial039.stdout:868: finishing write tid 3 to trial03942481-25 2026-02-25T08:45:27.320 INFO:tasks.rados.rados.0.trial039.stdout:868: finishing write tid 4 to trial03942481-25 2026-02-25T08:45:27.320 INFO:tasks.rados.rados.0.trial039.stdout:868: oid 25 updating version 0 to 39 2026-02-25T08:45:27.320 INFO:tasks.rados.rados.0.trial039.stdout:868: oid 25 version 39 is already newer than 39 2026-02-25T08:45:27.320 INFO:tasks.rados.rados.0.trial039.stdout:868: oid 25 version 39 is already newer than 39 2026-02-25T08:45:27.321 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 25 v 39 (ObjNum 419 snap 113 seq_num 419) dirty exists 2026-02-25T08:45:27.321 INFO:tasks.rados.rados.0.trial039.stdout:868: left oid 25 (ObjNum 419 snap 113 seq_num 419) 2026-02-25T08:45:27.355 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:27.358 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:27.359 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:27.999 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:28.078 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:28.076+0000 7f9c0fc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:28.081 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:28.080+0000 7f9c0fc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:28.109 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:28.127 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:28.134 INFO:tasks.rados.rados.0.trial039.stdout:868: done (2 left) 2026-02-25T08:45:28.134 INFO:tasks.rados.rados.0.trial039.stdout:869: done (1 left) 2026-02-25T08:45:28.135 INFO:tasks.rados.rados.0.trial039.stdout:870: done (0 left) 2026-02-25T08:45:28.135 INFO:tasks.rados.rados.0.trial039.stdout:871: read oid 31 snap 89 2026-02-25T08:45:28.135 INFO:tasks.rados.rados.0.trial039.stdout:871: expect deleted 2026-02-25T08:45:28.135 INFO:tasks.rados.rados.0.trial039.stdout:872: write oid 4 current snap is 114 2026-02-25T08:45:28.135 INFO:tasks.rados.rados.0.trial039.stdout:872: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2026-02-25T08:45:28.138 INFO:tasks.rados.rados.0.trial039.stdout:872: writing trial03942481-4 from 658940 to 1062935 tid 1 2026-02-25T08:45:28.145 INFO:tasks.rados.rados.0.trial039.stdout:872: writing trial03942481-4 from 1777496 to 2453632 tid 2 2026-02-25T08:45:28.147 INFO:tasks.rados.rados.0.trial039.stdout:872: writing trial03942481-4 from 2950951 to 3058940 tid 3 2026-02-25T08:45:28.147 INFO:tasks.rados.rados.0.trial039.stdout:873: write (excl) oid 33 current snap is 114 2026-02-25T08:45:28.147 INFO:tasks.rados.rados.0.trial039.stdout:873: seq_num 421 ranges {675747=479244,1835978=536147} 2026-02-25T08:45:28.152 INFO:tasks.rados.rados.0.trial039.stdout:873: writing trial03942481-33 from 675747 to 1154991 tid 1 2026-02-25T08:45:28.156 INFO:tasks.rados.rados.0.trial039.stdout:873: writing trial03942481-33 from 1835978 to 2372125 tid 2 2026-02-25T08:45:28.156 INFO:tasks.rados.rados.0.trial039.stdout:874: snap_remove snap 113 2026-02-25T08:45:29.141 INFO:tasks.rados.rados.0.trial039.stdout:874: done (3 left) 2026-02-25T08:45:29.141 INFO:tasks.rados.rados.0.trial039.stdout:875: read oid 50 snap -1 2026-02-25T08:45:29.141 INFO:tasks.rados.rados.0.trial039.stdout:875: expect (ObjNum 411 snap 111 seq_num 411) 2026-02-25T08:45:29.141 INFO:tasks.rados.rados.0.trial039.stdout:876: read oid 20 snap -1 2026-02-25T08:45:29.142 INFO:tasks.rados.rados.0.trial039.stdout:876: expect (ObjNum 305 snap 81 seq_num 305) 2026-02-25T08:45:29.142 INFO:tasks.rados.rados.0.trial039.stdout:877: snap_remove snap 106 2026-02-25T08:45:30.127 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:30.140 INFO:tasks.rados.rados.0.trial039.stdout:877: done (5 left) 2026-02-25T08:45:30.140 INFO:tasks.rados.rados.0.trial039.stdout:878: snap_create 2026-02-25T08:45:30.140 INFO:tasks.rados.rados.0.trial039.stdout:873: finishing write tid 1 to trial03942481-33 2026-02-25T08:45:30.140 INFO:tasks.rados.rados.0.trial039.stdout:872: finishing write tid 1 to trial03942481-4 2026-02-25T08:45:30.140 INFO:tasks.rados.rados.0.trial039.stdout:872: finishing write tid 2 to trial03942481-4 2026-02-25T08:45:30.140 INFO:tasks.rados.rados.0.trial039.stdout:873: finishing write tid 2 to trial03942481-33 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:872: finishing write tid 3 to trial03942481-4 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:872: finishing write tid 4 to trial03942481-4 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:873: finishing write tid 3 to trial03942481-33 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:872: finishing write tid 5 to trial03942481-4 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:872: oid 4 updating version 0 to 157 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:872: oid 4 version 157 is already newer than 157 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:872: oid 4 version 157 is already newer than 157 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:872: oid 4 version 157 is already newer than 157 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 157 (ObjNum 420 snap 114 seq_num 420) dirty exists 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:872: left oid 4 (ObjNum 420 snap 114 seq_num 420) 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:873: finishing write tid 4 to trial03942481-33 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:873: oid 33 updating version 0 to 132 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:873: oid 33 version 132 is already newer than 130 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:873: oid 33 version 132 is already newer than 132 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 33 v 132 (ObjNum 421 snap 114 seq_num 421) dirty exists 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:873: left oid 33 (ObjNum 421 snap 114 seq_num 421) 2026-02-25T08:45:30.141 INFO:tasks.rados.rados.0.trial039.stdout:875: expect (ObjNum 411 snap 111 seq_num 411) 2026-02-25T08:45:30.184 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:30.210 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:30.208+0000 7f1c0990b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:30.213 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:30.212+0000 7f1c0990b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:30.240 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:30.258 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:30.280 INFO:tasks.rados.rados.0.trial039.stdout:876: expect (ObjNum 305 snap 81 seq_num 305) 2026-02-25T08:45:30.352 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 935516 2026-02-25T08:45:31.147 INFO:tasks.rados.rados.0.trial039.stdout:871: done (5 left) 2026-02-25T08:45:31.147 INFO:tasks.rados.rados.0.trial039.stdout:872: done (4 left) 2026-02-25T08:45:31.147 INFO:tasks.rados.rados.0.trial039.stdout:873: done (3 left) 2026-02-25T08:45:31.147 INFO:tasks.rados.rados.0.trial039.stdout:875: done (2 left) 2026-02-25T08:45:31.147 INFO:tasks.rados.rados.0.trial039.stdout:876: done (1 left) 2026-02-25T08:45:31.147 INFO:tasks.rados.rados.0.trial039.stdout:878: done (0 left) 2026-02-25T08:45:31.147 INFO:tasks.rados.rados.0.trial039.stdout:879: write oid 29 current snap is 115 2026-02-25T08:45:31.147 INFO:tasks.rados.rados.0.trial039.stdout:879: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2026-02-25T08:45:31.152 INFO:tasks.rados.rados.0.trial039.stdout:879: writing trial03942481-29 from 692554 to 1247047 tid 1 2026-02-25T08:45:31.157 INFO:tasks.rados.rados.0.trial039.stdout:879: writing trial03942481-29 from 1778107 to 2457912 tid 2 2026-02-25T08:45:31.159 INFO:tasks.rados.rados.0.trial039.stdout:879: writing trial03942481-29 from 2889444 to 3092554 tid 3 2026-02-25T08:45:31.159 INFO:tasks.rados.rados.0.trial039.stdout:880: read oid 2 snap -1 2026-02-25T08:45:31.159 INFO:tasks.rados.rados.0.trial039.stdout:880: expect (ObjNum 407 snap 111 seq_num 407) 2026-02-25T08:45:31.159 INFO:tasks.rados.rados.0.trial039.stdout:881: read oid 46 snap -1 2026-02-25T08:45:31.159 INFO:tasks.rados.rados.0.trial039.stdout:881: expect deleted 2026-02-25T08:45:31.159 INFO:tasks.rados.rados.0.trial039.stdout:882: snap_remove snap 91 2026-02-25T08:45:32.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~237s 2026-02-25T08:45:32.152 INFO:tasks.rados.rados.0.trial039.stdout:882: done (3 left) 2026-02-25T08:45:32.152 INFO:tasks.rados.rados.0.trial039.stdout:883: read oid 36 snap -1 2026-02-25T08:45:32.152 INFO:tasks.rados.rados.0.trial039.stdout:883: expect deleted 2026-02-25T08:45:32.152 INFO:tasks.rados.rados.0.trial039.stdout:884: read oid 48 snap 96 2026-02-25T08:45:32.152 INFO:tasks.rados.rados.0.trial039.stdout:884: expect (ObjNum 329 snap 86 seq_num 329) 2026-02-25T08:45:32.153 INFO:tasks.rados.rados.0.trial039.stdout:885: read oid 40 snap -1 2026-02-25T08:45:32.153 INFO:tasks.rados.rados.0.trial039.stdout:885: expect (ObjNum 388 snap 106 seq_num 388) 2026-02-25T08:45:32.153 INFO:tasks.rados.rados.0.trial039.stdout:886: read oid 16 snap -1 2026-02-25T08:45:32.153 INFO:tasks.rados.rados.0.trial039.stdout:886: expect deleted 2026-02-25T08:45:32.153 INFO:tasks.rados.rados.0.trial039.stdout:887: read oid 47 snap -1 2026-02-25T08:45:32.153 INFO:tasks.rados.rados.0.trial039.stdout:887: expect (ObjNum 406 snap 111 seq_num 406) 2026-02-25T08:45:32.153 INFO:tasks.rados.rados.0.trial039.stdout:888: write oid 15 current snap is 115 2026-02-25T08:45:32.153 INFO:tasks.rados.rados.0.trial039.stdout:888: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2026-02-25T08:45:32.156 INFO:tasks.rados.rados.0.trial039.stdout:888: writing trial03942481-15 from 709361 to 1339103 tid 1 2026-02-25T08:45:32.160 INFO:tasks.rados.rados.0.trial039.stdout:888: writing trial03942481-15 from 1836589 to 2376405 tid 2 2026-02-25T08:45:32.163 INFO:tasks.rados.rados.0.trial039.stdout:888: writing trial03942481-15 from 3033220 to 3109361 tid 3 2026-02-25T08:45:32.163 INFO:tasks.rados.rados.0.trial039.stdout:889: write oid 44 current snap is 115 2026-02-25T08:45:32.163 INFO:tasks.rados.rados.0.trial039.stdout:889: seq_num 424 ranges {726168=704991,1895071=683474} 2026-02-25T08:45:32.168 INFO:tasks.rados.rados.0.trial039.stdout:889: writing trial03942481-44 from 726168 to 1431159 tid 1 2026-02-25T08:45:32.173 INFO:tasks.rados.rados.0.trial039.stdout:889: writing trial03942481-44 from 1895071 to 2578545 tid 2 2026-02-25T08:45:32.173 INFO:tasks.rados.rados.0.trial039.stdout:890: snap_create 2026-02-25T08:45:32.173 INFO:tasks.rados.rados.0.trial039.stdout:880: expect (ObjNum 407 snap 111 seq_num 407) 2026-02-25T08:45:32.244 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2222898 2026-02-25T08:45:32.259 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:32.262 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:32.314 INFO:tasks.rados.rados.0.trial039.stdout:879: finishing write tid 1 to trial03942481-29 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:879: finishing write tid 2 to trial03942481-29 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:879: finishing write tid 3 to trial03942481-29 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:879: finishing write tid 4 to trial03942481-29 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:879: finishing write tid 5 to trial03942481-29 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:879: oid 29 updating version 0 to 86 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:879: oid 29 version 86 is already newer than 86 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:879: oid 29 version 86 is already newer than 86 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:879: oid 29 version 86 is already newer than 86 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 29 v 86 (ObjNum 422 snap 115 seq_num 422) dirty exists 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:879: left oid 29 (ObjNum 422 snap 115 seq_num 422) 2026-02-25T08:45:32.315 INFO:tasks.rados.rados.0.trial039.stdout:885: expect (ObjNum 388 snap 106 seq_num 388) 2026-02-25T08:45:32.337 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:32.334+0000 7f359121a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:32.339 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:32.337+0000 7f359121a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:32.355 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1707507 2026-02-25T08:45:32.359 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:32.369 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:32.385 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:32.385 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:32.390 INFO:tasks.rados.rados.0.trial039.stdout:889: finishing write tid 1 to trial03942481-44 2026-02-25T08:45:32.390 INFO:tasks.rados.rados.0.trial039.stdout:888: finishing write tid 1 to trial03942481-15 2026-02-25T08:45:32.390 INFO:tasks.rados.rados.0.trial039.stdout:884: expect (ObjNum 329 snap 86 seq_num 329) 2026-02-25T08:45:32.432 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:32.435 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:32.435 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:32.453 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1751228 2026-02-25T08:45:32.471 INFO:tasks.rados.rados.0.trial039.stdout:887: expect (ObjNum 406 snap 111 seq_num 406) 2026-02-25T08:45:32.497 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1499541 2026-02-25T08:45:32.520 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2373220 2026-02-25T08:45:32.525 INFO:tasks.rados.rados.0.trial039.stdout:889: finishing write tid 2 to trial03942481-44 2026-02-25T08:45:32.525 INFO:tasks.rados.rados.0.trial039.stdout:888: finishing write tid 2 to trial03942481-15 2026-02-25T08:45:32.525 INFO:tasks.rados.rados.0.trial039.stdout:889: finishing write tid 3 to trial03942481-44 2026-02-25T08:45:32.525 INFO:tasks.rados.rados.0.trial039.stdout:889: finishing write tid 4 to trial03942481-44 2026-02-25T08:45:32.525 INFO:tasks.rados.rados.0.trial039.stdout:889: oid 44 updating version 0 to 170 2026-02-25T08:45:32.525 INFO:tasks.rados.rados.0.trial039.stdout:889: oid 44 version 170 is already newer than 170 2026-02-25T08:45:32.525 INFO:tasks.rados.rados.0.trial039.stdout:889: oid 44 version 170 is already newer than 170 2026-02-25T08:45:32.525 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 170 (ObjNum 424 snap 115 seq_num 424) dirty exists 2026-02-25T08:45:32.525 INFO:tasks.rados.rados.0.trial039.stdout:889: left oid 44 (ObjNum 424 snap 115 seq_num 424) 2026-02-25T08:45:32.526 INFO:tasks.rados.rados.0.trial039.stdout:888: finishing write tid 3 to trial03942481-15 2026-02-25T08:45:32.526 INFO:tasks.rados.rados.0.trial039.stdout:888: finishing write tid 4 to trial03942481-15 2026-02-25T08:45:32.526 INFO:tasks.rados.rados.0.trial039.stdout:888: finishing write tid 5 to trial03942481-15 2026-02-25T08:45:32.526 INFO:tasks.rados.rados.0.trial039.stdout:888: oid 15 updating version 0 to 161 2026-02-25T08:45:32.526 INFO:tasks.rados.rados.0.trial039.stdout:888: oid 15 version 161 is already newer than 161 2026-02-25T08:45:32.526 INFO:tasks.rados.rados.0.trial039.stdout:888: oid 15 version 161 is already newer than 161 2026-02-25T08:45:32.526 INFO:tasks.rados.rados.0.trial039.stdout:888: oid 15 version 161 is already newer than 161 2026-02-25T08:45:32.526 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 15 v 161 (ObjNum 423 snap 115 seq_num 423) dirty exists 2026-02-25T08:45:32.526 INFO:tasks.rados.rados.0.trial039.stdout:888: left oid 15 (ObjNum 423 snap 115 seq_num 423) 2026-02-25T08:45:33.088 INFO:tasks.rados.rados.0.trial039.stdout:879: done (10 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:880: done (9 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:881: done (8 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:883: done (7 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:884: done (6 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:885: done (5 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:886: done (4 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:887: done (3 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:888: done (2 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:889: done (1 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:890: done (0 left) 2026-02-25T08:45:33.089 INFO:tasks.rados.rados.0.trial039.stdout:891: write oid 32 current snap is 116 2026-02-25T08:45:33.090 INFO:tasks.rados.rados.0.trial039.stdout:891: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2026-02-25T08:45:33.094 INFO:tasks.rados.rados.0.trial039.stdout:891: writing trial03942481-32 from 742975 to 1523215 tid 1 2026-02-25T08:45:33.097 INFO:tasks.rados.rados.0.trial039.stdout:891: writing trial03942481-32 from 1953553 to 2380685 tid 2 2026-02-25T08:45:33.100 INFO:tasks.rados.rados.0.trial039.stdout:891: writing trial03942481-32 from 2971713 to 3142975 tid 3 2026-02-25T08:45:33.100 INFO:tasks.rados.rados.0.trial039.stdout:892: delete oid 34 current snap is 116 2026-02-25T08:45:33.147 INFO:tasks.rados.rados.0.trial039.stdout:891: finishing write tid 1 to trial03942481-32 2026-02-25T08:45:33.161 INFO:tasks.rados.rados.0.trial039.stdout:891: finishing write tid 2 to trial03942481-32 2026-02-25T08:45:33.175 INFO:tasks.rados.rados.0.trial039.stdout:891: finishing write tid 3 to trial03942481-32 2026-02-25T08:45:33.183 INFO:tasks.rados.rados.0.trial039.stdout:891: finishing write tid 4 to trial03942481-32 2026-02-25T08:45:33.186 INFO:tasks.rados.rados.0.trial039.stdout:891: finishing write tid 5 to trial03942481-32 2026-02-25T08:45:33.186 INFO:tasks.rados.rados.0.trial039.stdout:891: oid 32 updating version 0 to 111 2026-02-25T08:45:33.186 INFO:tasks.rados.rados.0.trial039.stdout:891: oid 32 version 111 is already newer than 111 2026-02-25T08:45:33.187 INFO:tasks.rados.rados.0.trial039.stdout:891: oid 32 version 111 is already newer than 111 2026-02-25T08:45:33.187 INFO:tasks.rados.rados.0.trial039.stdout:891: oid 32 version 111 is already newer than 111 2026-02-25T08:45:33.187 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 32 v 111 (ObjNum 425 snap 116 seq_num 425) dirty exists 2026-02-25T08:45:33.187 INFO:tasks.rados.rados.0.trial039.stdout:891: left oid 32 (ObjNum 425 snap 116 seq_num 425) 2026-02-25T08:45:33.198 INFO:tasks.rados.rados.0.trial039.stdout:891: done (1 left) 2026-02-25T08:45:33.198 INFO:tasks.rados.rados.0.trial039.stdout:892: done (0 left) 2026-02-25T08:45:33.198 INFO:tasks.rados.rados.0.trial039.stdout:893: read oid 37 snap -1 2026-02-25T08:45:33.199 INFO:tasks.rados.rados.0.trial039.stdout:893: expect deleted 2026-02-25T08:45:33.199 INFO:tasks.rados.rados.0.trial039.stdout:894: delete oid 4 current snap is 116 2026-02-25T08:45:33.225 INFO:tasks.rados.rados.0.trial039.stdout:893: done (1 left) 2026-02-25T08:45:33.226 INFO:tasks.rados.rados.0.trial039.stdout:894: done (0 left) 2026-02-25T08:45:33.226 INFO:tasks.rados.rados.0.trial039.stdout:895: read oid 32 snap -1 2026-02-25T08:45:33.226 INFO:tasks.rados.rados.0.trial039.stdout:895: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-25T08:45:33.226 INFO:tasks.rados.rados.0.trial039.stdout:896: read oid 46 snap 108 2026-02-25T08:45:33.226 INFO:tasks.rados.rados.0.trial039.stdout:896: expect (ObjNum 393 snap 108 seq_num 393) 2026-02-25T08:45:33.227 INFO:tasks.rados.rados.0.trial039.stdout:897: write oid 10 current snap is 116 2026-02-25T08:45:33.227 INFO:tasks.rados.rados.0.trial039.stdout:897: seq_num 428 ranges {793396=722340,2128999=690812} 2026-02-25T08:45:33.230 INFO:tasks.rados.rados.0.trial039.stdout:897: writing trial03942481-10 from 793396 to 1515736 tid 1 2026-02-25T08:45:33.235 INFO:tasks.rados.rados.0.trial039.stdout:897: writing trial03942481-10 from 2128999 to 2819811 tid 2 2026-02-25T08:45:33.236 INFO:tasks.rados.rados.0.trial039.stdout:898: delete oid 16 current snap is 116 2026-02-25T08:45:33.238 INFO:tasks.rados.rados.0.trial039.stdout:896: expect (ObjNum 393 snap 108 seq_num 393) 2026-02-25T08:45:33.253 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2013911 2026-02-25T08:45:33.284 INFO:tasks.rados.rados.0.trial039.stdout:896: done (3 left) 2026-02-25T08:45:33.284 INFO:tasks.rados.rados.0.trial039.stdout:898: done (2 left) 2026-02-25T08:45:33.284 INFO:tasks.rados.rados.0.trial039.stdout:899: snap_remove snap 98 2026-02-25T08:45:34.158 INFO:tasks.rados.rados.0.trial039.stdout:899: done (2 left) 2026-02-25T08:45:34.159 INFO:tasks.rados.rados.0.trial039.stdout:900: read oid 42 snap -1 2026-02-25T08:45:34.159 INFO:tasks.rados.rados.0.trial039.stdout:900: expect (ObjNum 359 snap 97 seq_num 359) 2026-02-25T08:45:34.159 INFO:tasks.rados.rados.0.trial039.stdout:901: read oid 21 snap -1 2026-02-25T08:45:34.159 INFO:tasks.rados.rados.0.trial039.stdout:901: expect deleted 2026-02-25T08:45:34.159 INFO:tasks.rados.rados.0.trial039.stdout:902: delete oid 26 current snap is 116 2026-02-25T08:45:34.159 INFO:tasks.rados.rados.0.trial039.stdout:895: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-25T08:45:34.174 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2213745 2026-02-25T08:45:34.202 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:34.265 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1156457 2026-02-25T08:45:34.306 INFO:tasks.rados.rados.0.trial039.stdout:897: finishing write tid 1 to trial03942481-10 2026-02-25T08:45:34.306 INFO:tasks.rados.rados.0.trial039.stdout:897: finishing write tid 2 to trial03942481-10 2026-02-25T08:45:34.306 INFO:tasks.rados.rados.0.trial039.stdout:897: finishing write tid 3 to trial03942481-10 2026-02-25T08:45:34.306 INFO:tasks.rados.rados.0.trial039.stdout:895: done (4 left) 2026-02-25T08:45:34.306 INFO:tasks.rados.rados.0.trial039.stdout:902: done (3 left) 2026-02-25T08:45:34.306 INFO:tasks.rados.rados.0.trial039.stdout:903: write oid 7 current snap is 116 2026-02-25T08:45:34.306 INFO:tasks.rados.rados.0.trial039.stdout:897: finishing write tid 4 to trial03942481-10 2026-02-25T08:45:34.307 INFO:tasks.rados.rados.0.trial039.stdout:897: oid 10 updating version 0 to 136 2026-02-25T08:45:34.307 INFO:tasks.rados.rados.0.trial039.stdout:897: oid 10 version 136 is already newer than 136 2026-02-25T08:45:34.307 INFO:tasks.rados.rados.0.trial039.stdout:897: oid 10 version 136 is already newer than 136 2026-02-25T08:45:34.307 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 10 v 136 (ObjNum 428 snap 116 seq_num 428) dirty exists 2026-02-25T08:45:34.307 INFO:tasks.rados.rados.0.trial039.stdout:897: left oid 10 (ObjNum 428 snap 116 seq_num 428) 2026-02-25T08:45:34.307 INFO:tasks.rados.rados.0.trial039.stdout:903: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2026-02-25T08:45:34.308 INFO:tasks.rados.rados.0.trial039.stdout:903: writing trial03942481-7 from 443817 to 991904 tid 1 2026-02-25T08:45:34.313 INFO:tasks.rados.rados.0.trial039.stdout:903: writing trial03942481-7 from 1788092 to 2226231 tid 2 2026-02-25T08:45:34.318 INFO:tasks.rados.rados.0.trial039.stdout:903: writing trial03942481-7 from 3019898 to 3243817 tid 3 2026-02-25T08:45:34.318 INFO:tasks.rados.rados.0.trial039.stdout:897: done (3 left) 2026-02-25T08:45:34.318 INFO:tasks.rados.rados.0.trial039.stdout:904: read oid 15 snap 112 2026-02-25T08:45:34.318 INFO:tasks.rados.rados.0.trial039.stdout:904: expect deleted 2026-02-25T08:45:34.318 INFO:tasks.rados.rados.0.trial039.stdout:905: read oid 25 snap 108 2026-02-25T08:45:34.318 INFO:tasks.rados.rados.0.trial039.stdout:905: expect (ObjNum 367 snap 98 seq_num 367) 2026-02-25T08:45:34.318 INFO:tasks.rados.rados.0.trial039.stdout:906: read oid 46 snap 96 2026-02-25T08:45:34.318 INFO:tasks.rados.rados.0.trial039.stdout:906: expect (ObjNum 351 snap 94 seq_num 351) 2026-02-25T08:45:34.319 INFO:tasks.rados.rados.0.trial039.stdout:907: read oid 34 snap -1 2026-02-25T08:45:34.319 INFO:tasks.rados.rados.0.trial039.stdout:907: expect deleted 2026-02-25T08:45:34.319 INFO:tasks.rados.rados.0.trial039.stdout:908: write (excl) oid 44 current snap is 116 2026-02-25T08:45:34.319 INFO:tasks.rados.rados.0.trial039.stdout:908: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2026-02-25T08:45:34.321 INFO:tasks.rados.rados.0.trial039.stdout:908: writing trial03942481-44 from 460624 to 1083960 tid 1 2026-02-25T08:45:34.327 INFO:tasks.rados.rados.0.trial039.stdout:908: writing trial03942481-44 from 1846574 to 2544724 tid 2 2026-02-25T08:45:34.329 INFO:tasks.rados.rados.0.trial039.stdout:908: writing trial03942481-44 from 3163674 to 3260624 tid 3 2026-02-25T08:45:34.330 INFO:tasks.rados.rados.0.trial039.stdout:909: write oid 4 current snap is 116 2026-02-25T08:45:34.330 INFO:tasks.rados.rados.0.trial039.stdout:909: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2026-02-25T08:45:34.334 INFO:tasks.rados.rados.0.trial039.stdout:909: writing trial03942481-4 from 477431 to 1176016 tid 1 2026-02-25T08:45:34.338 INFO:tasks.rados.rados.0.trial039.stdout:909: writing trial03942481-4 from 1905056 to 2346864 tid 2 2026-02-25T08:45:34.342 INFO:tasks.rados.rados.0.trial039.stdout:909: writing trial03942481-4 from 3074744 to 3277431 tid 3 2026-02-25T08:45:34.342 INFO:tasks.rados.rados.0.trial039.stdout:910: read oid 29 snap -1 2026-02-25T08:45:34.342 INFO:tasks.rados.rados.0.trial039.stdout:910: expect (ObjNum 422 snap 115 seq_num 422) 2026-02-25T08:45:34.342 INFO:tasks.rados.rados.0.trial039.stdout:911: write (excl) oid 40 current snap is 116 2026-02-25T08:45:34.342 INFO:tasks.rados.rados.0.trial039.stdout:911: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2026-02-25T08:45:34.345 INFO:tasks.rados.rados.0.trial039.stdout:911: writing trial03942481-40 from 494238 to 984425 tid 1 2026-02-25T08:45:34.350 INFO:tasks.rados.rados.0.trial039.stdout:911: writing trial03942481-40 from 1563538 to 2265357 tid 2 2026-02-25T08:45:34.354 INFO:tasks.rados.rados.0.trial039.stdout:911: writing trial03942481-40 from 2818520 to 3294238 tid 3 2026-02-25T08:45:34.355 INFO:tasks.rados.rados.0.trial039.stdout:912: snap_remove snap 100 2026-02-25T08:45:34.386 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:34.465 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:34.462+0000 7fb0a10cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:34.467 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:34.465+0000 7fb0a10cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:34.494 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:34.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:35.163 INFO:tasks.rados.rados.0.trial039.stdout:912: done (11 left) 2026-02-25T08:45:35.163 INFO:tasks.rados.rados.0.trial039.stdout:913: delete oid 16 current snap is 116 2026-02-25T08:45:35.164 INFO:tasks.rados.rados.0.trial039.stdout:900: expect (ObjNum 359 snap 97 seq_num 359) 2026-02-25T08:45:35.195 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1580297 2026-02-25T08:45:35.198 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:35.220 INFO:tasks.rados.rados.0.trial039.stdout:900: done (11 left) 2026-02-25T08:45:35.220 INFO:tasks.rados.rados.0.trial039.stdout:901: done (10 left) 2026-02-25T08:45:35.221 INFO:tasks.rados.rados.0.trial039.stdout:913: done (9 left) 2026-02-25T08:45:35.221 INFO:tasks.rados.rados.0.trial039.stdout:914: read oid 38 snap -1 2026-02-25T08:45:35.221 INFO:tasks.rados.rados.0.trial039.stdout:914: expect (ObjNum 408 snap 111 seq_num 408) 2026-02-25T08:45:35.221 INFO:tasks.rados.rados.0.trial039.stdout:915: read oid 32 snap -1 2026-02-25T08:45:35.221 INFO:tasks.rados.rados.0.trial039.stdout:915: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-25T08:45:35.221 INFO:tasks.rados.rados.0.trial039.stdout:916: snap_create 2026-02-25T08:45:35.221 INFO:tasks.rados.rados.0.trial039.stdout:903: finishing write tid 1 to trial03942481-7 2026-02-25T08:45:35.221 INFO:tasks.rados.rados.0.trial039.stdout:903: finishing write tid 2 to trial03942481-7 2026-02-25T08:45:35.222 INFO:tasks.rados.rados.0.trial039.stdout:903: finishing write tid 3 to trial03942481-7 2026-02-25T08:45:35.222 INFO:tasks.rados.rados.0.trial039.stdout:903: finishing write tid 4 to trial03942481-7 2026-02-25T08:45:35.222 INFO:tasks.rados.rados.0.trial039.stdout:903: finishing write tid 5 to trial03942481-7 2026-02-25T08:45:35.222 INFO:tasks.rados.rados.0.trial039.stdout:903: oid 7 updating version 0 to 174 2026-02-25T08:45:35.222 INFO:tasks.rados.rados.0.trial039.stdout:903: oid 7 version 174 is already newer than 174 2026-02-25T08:45:35.222 INFO:tasks.rados.rados.0.trial039.stdout:903: oid 7 version 174 is already newer than 174 2026-02-25T08:45:35.222 INFO:tasks.rados.rados.0.trial039.stdout:903: oid 7 version 174 is already newer than 174 2026-02-25T08:45:35.222 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 7 v 174 (ObjNum 431 snap 116 seq_num 431) dirty exists 2026-02-25T08:45:35.222 INFO:tasks.rados.rados.0.trial039.stdout:903: left oid 7 (ObjNum 431 snap 116 seq_num 431) 2026-02-25T08:45:35.222 INFO:tasks.rados.rados.0.trial039.stdout:906: expect (ObjNum 351 snap 94 seq_num 351) 2026-02-25T08:45:35.246 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2314636 2026-02-25T08:45:35.382 INFO:tasks.rados.rados.0.trial039.stdout:909: finishing write tid 1 to trial03942481-4 2026-02-25T08:45:35.382 INFO:tasks.rados.rados.0.trial039.stdout:908: finishing write tid 1 to trial03942481-44 2026-02-25T08:45:35.382 INFO:tasks.rados.rados.0.trial039.stdout:909: finishing write tid 2 to trial03942481-4 2026-02-25T08:45:35.382 INFO:tasks.rados.rados.0.trial039.stdout:911: finishing write tid 1 to trial03942481-40 2026-02-25T08:45:35.383 INFO:tasks.rados.rados.0.trial039.stdout:905: expect (ObjNum 367 snap 98 seq_num 367) 2026-02-25T08:45:35.383 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:35.413 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:35.477 INFO:tasks.rados.rados.0.trial039.stdout:908: finishing write tid 2 to trial03942481-44 2026-02-25T08:45:35.477 INFO:tasks.rados.rados.0.trial039.stdout:909: finishing write tid 3 to trial03942481-4 2026-02-25T08:45:35.477 INFO:tasks.rados.rados.0.trial039.stdout:908: finishing write tid 3 to trial03942481-44 2026-02-25T08:45:35.477 INFO:tasks.rados.rados.0.trial039.stdout:909: finishing write tid 4 to trial03942481-4 2026-02-25T08:45:35.478 INFO:tasks.rados.rados.0.trial039.stdout:909: finishing write tid 5 to trial03942481-4 2026-02-25T08:45:35.478 INFO:tasks.rados.rados.0.trial039.stdout:909: oid 4 updating version 0 to 167 2026-02-25T08:45:35.478 INFO:tasks.rados.rados.0.trial039.stdout:909: oid 4 version 167 is already newer than 167 2026-02-25T08:45:35.478 INFO:tasks.rados.rados.0.trial039.stdout:909: oid 4 version 167 is already newer than 167 2026-02-25T08:45:35.478 INFO:tasks.rados.rados.0.trial039.stdout:909: oid 4 version 167 is already newer than 167 2026-02-25T08:45:35.478 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 4 v 167 (ObjNum 433 snap 116 seq_num 433) dirty exists 2026-02-25T08:45:35.478 INFO:tasks.rados.rados.0.trial039.stdout:909: left oid 4 (ObjNum 433 snap 116 seq_num 433) 2026-02-25T08:45:35.478 INFO:tasks.rados.rados.0.trial039.stdout:911: finishing write tid 2 to trial03942481-40 2026-02-25T08:45:35.478 INFO:tasks.rados.rados.0.trial039.stdout:908: finishing write tid 4 to trial03942481-44 2026-02-25T08:45:35.479 INFO:tasks.rados.rados.0.trial039.stdout:908: finishing write tid 5 to trial03942481-44 2026-02-25T08:45:35.479 INFO:tasks.rados.rados.0.trial039.stdout:908: oid 44 updating version 0 to 179 2026-02-25T08:45:35.479 INFO:tasks.rados.rados.0.trial039.stdout:908: oid 44 version 179 is already newer than 176 2026-02-25T08:45:35.479 INFO:tasks.rados.rados.0.trial039.stdout:908: oid 44 version 179 is already newer than 179 2026-02-25T08:45:35.479 INFO:tasks.rados.rados.0.trial039.stdout:908: oid 44 version 179 is already newer than 179 2026-02-25T08:45:35.479 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 44 v 179 (ObjNum 432 snap 116 seq_num 432) dirty exists 2026-02-25T08:45:35.479 INFO:tasks.rados.rados.0.trial039.stdout:908: left oid 44 (ObjNum 432 snap 116 seq_num 432) 2026-02-25T08:45:35.479 INFO:tasks.rados.rados.0.trial039.stdout:910: expect (ObjNum 422 snap 115 seq_num 422) 2026-02-25T08:45:35.479 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:35.599 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:35.660 INFO:tasks.rados.rados.0.trial039.stdout:911: finishing write tid 3 to trial03942481-40 2026-02-25T08:45:35.661 INFO:tasks.rados.rados.0.trial039.stdout:911: finishing write tid 4 to trial03942481-40 2026-02-25T08:45:35.661 INFO:tasks.rados.rados.0.trial039.stdout:911: finishing write tid 5 to trial03942481-40 2026-02-25T08:45:35.661 INFO:tasks.rados.rados.0.trial039.stdout:911: oid 40 updating version 0 to 187 2026-02-25T08:45:35.661 INFO:tasks.rados.rados.0.trial039.stdout:911: oid 40 version 187 is already newer than 184 2026-02-25T08:45:35.661 INFO:tasks.rados.rados.0.trial039.stdout:911: oid 40 version 187 is already newer than 187 2026-02-25T08:45:35.661 INFO:tasks.rados.rados.0.trial039.stdout:911: oid 40 version 187 is already newer than 187 2026-02-25T08:45:35.661 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 40 v 187 (ObjNum 434 snap 116 seq_num 434) dirty exists 2026-02-25T08:45:35.661 INFO:tasks.rados.rados.0.trial039.stdout:911: left oid 40 (ObjNum 434 snap 116 seq_num 434) 2026-02-25T08:45:35.661 INFO:tasks.rados.rados.0.trial039.stdout:914: expect (ObjNum 408 snap 111 seq_num 408) 2026-02-25T08:45:35.731 INFO:tasks.rados.rados.0.trial039.stdout:915: expect (ObjNum 425 snap 116 seq_num 425) 2026-02-25T08:45:35.840 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:36.167 INFO:tasks.rados.rados.0.trial039.stdout:903: done (11 left) 2026-02-25T08:45:36.167 INFO:tasks.rados.rados.0.trial039.stdout:904: done (10 left) 2026-02-25T08:45:36.167 INFO:tasks.rados.rados.0.trial039.stdout:905: done (9 left) 2026-02-25T08:45:36.167 INFO:tasks.rados.rados.0.trial039.stdout:906: done (8 left) 2026-02-25T08:45:36.167 INFO:tasks.rados.rados.0.trial039.stdout:907: done (7 left) 2026-02-25T08:45:36.167 INFO:tasks.rados.rados.0.trial039.stdout:908: done (6 left) 2026-02-25T08:45:36.167 INFO:tasks.rados.rados.0.trial039.stdout:909: done (5 left) 2026-02-25T08:45:36.168 INFO:tasks.rados.rados.0.trial039.stdout:910: done (4 left) 2026-02-25T08:45:36.168 INFO:tasks.rados.rados.0.trial039.stdout:911: done (3 left) 2026-02-25T08:45:36.168 INFO:tasks.rados.rados.0.trial039.stdout:914: done (2 left) 2026-02-25T08:45:36.168 INFO:tasks.rados.rados.0.trial039.stdout:915: done (1 left) 2026-02-25T08:45:36.168 INFO:tasks.rados.rados.0.trial039.stdout:916: done (0 left) 2026-02-25T08:45:36.168 INFO:tasks.rados.rados.0.trial039.stdout:917: delete oid 45 current snap is 117 2026-02-25T08:45:36.181 INFO:tasks.rados.rados.0.trial039.stdout:917: done (0 left) 2026-02-25T08:45:36.181 INFO:tasks.rados.rados.0.trial039.stdout:918: read oid 27 snap -1 2026-02-25T08:45:36.181 INFO:tasks.rados.rados.0.trial039.stdout:918: expect (ObjNum 325 snap 85 seq_num 325) 2026-02-25T08:45:36.181 INFO:tasks.rados.rados.0.trial039.stdout:919: delete oid 2 current snap is 117 2026-02-25T08:45:36.209 INFO:tasks.rados.rados.0.trial039.stdout:918: expect (ObjNum 325 snap 85 seq_num 325) 2026-02-25T08:45:36.246 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:36.284 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:36.322 INFO:tasks.rados.rados.0.trial039.stdout:918: done (1 left) 2026-02-25T08:45:36.322 INFO:tasks.rados.rados.0.trial039.stdout:919: done (0 left) 2026-02-25T08:45:36.322 INFO:tasks.rados.rados.0.trial039.stdout:920: write (excl) oid 31 current snap is 117 2026-02-25T08:45:36.322 INFO:tasks.rados.rados.0.trial039.stdout:920: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2026-02-25T08:45:36.327 INFO:tasks.rados.rados.0.trial039.stdout:920: writing trial03942481-31 from 561466 to 1352649 tid 1 2026-02-25T08:45:36.332 INFO:tasks.rados.rados.0.trial039.stdout:920: writing trial03942481-31 from 2081113 to 2673917 tid 2 2026-02-25T08:45:36.335 INFO:tasks.rados.rados.0.trial039.stdout:920: writing trial03942481-31 from 3095506 to 3361466 tid 3 2026-02-25T08:45:36.335 INFO:tasks.rados.rados.0.trial039.stdout:921: snap_remove snap 96 2026-02-25T08:45:36.511 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:36.589 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:36.585+0000 7f7370bd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:36.591 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:36.589+0000 7f7370bd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:36.618 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:36.634 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:37.171 INFO:tasks.rados.rados.0.trial039.stdout:921: done (1 left) 2026-02-25T08:45:37.171 INFO:tasks.rados.rados.0.trial039.stdout:922: delete oid 45 current snap is 117 2026-02-25T08:45:37.171 INFO:tasks.rados.rados.0.trial039.stdout:920: finishing write tid 1 to trial03942481-31 2026-02-25T08:45:37.171 INFO:tasks.rados.rados.0.trial039.stdout:920: finishing write tid 2 to trial03942481-31 2026-02-25T08:45:37.172 INFO:tasks.rados.rados.0.trial039.stdout:920: finishing write tid 3 to trial03942481-31 2026-02-25T08:45:37.172 INFO:tasks.rados.rados.0.trial039.stdout:920: finishing write tid 4 to trial03942481-31 2026-02-25T08:45:37.172 INFO:tasks.rados.rados.0.trial039.stdout:920: finishing write tid 5 to trial03942481-31 2026-02-25T08:45:37.172 INFO:tasks.rados.rados.0.trial039.stdout:920: oid 31 updating version 0 to 320 2026-02-25T08:45:37.172 INFO:tasks.rados.rados.0.trial039.stdout:920: oid 31 version 320 is already newer than 320 2026-02-25T08:45:37.172 INFO:tasks.rados.rados.0.trial039.stdout:920: oid 31 version 320 is already newer than 317 2026-02-25T08:45:37.172 INFO:tasks.rados.rados.0.trial039.stdout:920: oid 31 version 320 is already newer than 320 2026-02-25T08:45:37.172 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 320 (ObjNum 438 snap 117 seq_num 438) dirty exists 2026-02-25T08:45:37.172 INFO:tasks.rados.rados.0.trial039.stdout:920: left oid 31 (ObjNum 438 snap 117 seq_num 438) 2026-02-25T08:45:37.186 INFO:tasks.rados.rados.0.trial039.stdout:920: done (1 left) 2026-02-25T08:45:37.186 INFO:tasks.rados.rados.0.trial039.stdout:922: done (0 left) 2026-02-25T08:45:37.186 INFO:tasks.rados.rados.0.trial039.stdout:923: write (excl) oid 27 current snap is 117 2026-02-25T08:45:37.187 INFO:tasks.rados.rados.0.trial039.stdout:923: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2026-02-25T08:45:37.190 INFO:tasks.rados.rados.0.trial039.stdout:923: writing trial03942481-27 from 595080 to 1136761 tid 1 2026-02-25T08:45:37.196 INFO:tasks.rados.rados.0.trial039.stdout:923: writing trial03942481-27 from 1798077 to 2394550 tid 2 2026-02-25T08:45:37.200 INFO:tasks.rados.rados.0.trial039.stdout:923: writing trial03942481-27 from 3150352 to 3395080 tid 3 2026-02-25T08:45:37.200 INFO:tasks.rados.rados.0.trial039.stdout:924: delete oid 49 current snap is 117 2026-02-25T08:45:37.225 INFO:tasks.rados.rados.0.trial039.stdout:923: finishing write tid 1 to trial03942481-27 2026-02-25T08:45:37.241 INFO:tasks.rados.rados.0.trial039.stdout:924: done (1 left) 2026-02-25T08:45:37.241 INFO:tasks.rados.rados.0.trial039.stdout:925: delete oid 21 current snap is 117 2026-02-25T08:45:37.245 INFO:tasks.rados.rados.0.trial039.stdout:923: finishing write tid 2 to trial03942481-27 2026-02-25T08:45:37.257 INFO:tasks.rados.rados.0.trial039.stdout:923: finishing write tid 3 to trial03942481-27 2026-02-25T08:45:37.264 INFO:tasks.rados.rados.0.trial039.stdout:925: done (1 left) 2026-02-25T08:45:37.265 INFO:tasks.rados.rados.0.trial039.stdout:926: snap_create 2026-02-25T08:45:37.267 INFO:tasks.rados.rados.0.trial039.stdout:923: finishing write tid 4 to trial03942481-27 2026-02-25T08:45:37.269 INFO:tasks.rados.rados.0.trial039.stdout:923: finishing write tid 5 to trial03942481-27 2026-02-25T08:45:37.269 INFO:tasks.rados.rados.0.trial039.stdout:923: oid 27 updating version 0 to 116 2026-02-25T08:45:37.269 INFO:tasks.rados.rados.0.trial039.stdout:923: oid 27 version 116 is already newer than 116 2026-02-25T08:45:37.269 INFO:tasks.rados.rados.0.trial039.stdout:923: oid 27 version 116 is already newer than 113 2026-02-25T08:45:37.269 INFO:tasks.rados.rados.0.trial039.stdout:923: oid 27 version 116 is already newer than 116 2026-02-25T08:45:37.270 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 27 v 116 (ObjNum 440 snap 117 seq_num 440) dirty exists 2026-02-25T08:45:37.270 INFO:tasks.rados.rados.0.trial039.stdout:923: left oid 27 (ObjNum 440 snap 117 seq_num 440) 2026-02-25T08:45:37.437 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:37.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~242s 2026-02-25T08:45:37.510 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:37.514 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:38.175 INFO:tasks.rados.rados.0.trial039.stdout:923: done (1 left) 2026-02-25T08:45:38.175 INFO:tasks.rados.rados.0.trial039.stdout:926: done (0 left) 2026-02-25T08:45:38.175 INFO:tasks.rados.rados.0.trial039.stdout:927: snap_create 2026-02-25T08:45:38.635 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:38.712 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:38.709+0000 7f57df4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:38.714 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:38.712+0000 7f57df4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:38.741 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:38.756 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:38.757 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:39.176 INFO:tasks.rados.rados.0.trial039.stdout:927: done (0 left) 2026-02-25T08:45:39.176 INFO:tasks.rados.rados.0.trial039.stdout:928: read oid 20 snap -1 2026-02-25T08:45:39.176 INFO:tasks.rados.rados.0.trial039.stdout:928: expect (ObjNum 305 snap 81 seq_num 305) 2026-02-25T08:45:39.176 INFO:tasks.rados.rados.0.trial039.stdout:929: write oid 6 current snap is 119 2026-02-25T08:45:39.176 INFO:tasks.rados.rados.0.trial039.stdout:929: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2026-02-25T08:45:39.180 INFO:tasks.rados.rados.0.trial039.stdout:929: writing trial03942481-6 from 645501 to 1129282 tid 1 2026-02-25T08:45:39.184 INFO:tasks.rados.rados.0.trial039.stdout:929: writing trial03942481-6 from 1573523 to 2033676 tid 2 2026-02-25T08:45:39.189 INFO:tasks.rados.rados.0.trial039.stdout:929: writing trial03942481-6 from 2548974 to 3312358 tid 3 2026-02-25T08:45:39.189 INFO:tasks.rados.rados.0.trial039.stdout:930: delete oid 48 current snap is 119 2026-02-25T08:45:39.214 INFO:tasks.rados.rados.0.trial039.stdout:928: expect (ObjNum 305 snap 81 seq_num 305) 2026-02-25T08:45:39.332 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:39.396 INFO:tasks.rados.rados.0.trial039.stdout:928: done (2 left) 2026-02-25T08:45:39.396 INFO:tasks.rados.rados.0.trial039.stdout:930: done (1 left) 2026-02-25T08:45:39.396 INFO:tasks.rados.rados.0.trial039.stdout:931: write (excl) oid 39 current snap is 119 2026-02-25T08:45:39.396 INFO:tasks.rados.rados.0.trial039.stdout:929: finishing write tid 1 to trial03942481-6 2026-02-25T08:45:39.396 INFO:tasks.rados.rados.0.trial039.stdout:931: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2026-02-25T08:45:39.401 INFO:tasks.rados.rados.0.trial039.stdout:931: writing trial03942481-39 from 679115 to 1313394 tid 1 2026-02-25T08:45:39.406 INFO:tasks.rados.rados.0.trial039.stdout:931: writing trial03942481-39 from 2090487 to 2554309 tid 2 2026-02-25T08:45:39.410 INFO:tasks.rados.rados.0.trial039.stdout:931: writing trial03942481-39 from 3003820 to 3479115 tid 3 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:932: delete oid 3 current snap is 119 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:929: finishing write tid 2 to trial03942481-6 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:929: finishing write tid 3 to trial03942481-6 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:929: finishing write tid 4 to trial03942481-6 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:929: finishing write tid 5 to trial03942481-6 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:929: oid 6 updating version 0 to 330 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:929: oid 6 version 330 is already newer than 330 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:929: oid 6 version 330 is already newer than 330 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:929: oid 6 version 330 is already newer than 330 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 6 v 330 (ObjNum 443 snap 119 seq_num 443) dirty exists 2026-02-25T08:45:39.411 INFO:tasks.rados.rados.0.trial039.stdout:929: left oid 6 (ObjNum 443 snap 119 seq_num 443) 2026-02-25T08:45:39.422 INFO:tasks.rados.rados.0.trial039.stdout:931: finishing write tid 1 to trial03942481-39 2026-02-25T08:45:39.448 INFO:tasks.rados.rados.0.trial039.stdout:931: finishing write tid 2 to trial03942481-39 2026-02-25T08:45:39.462 INFO:tasks.rados.rados.0.trial039.stdout:931: finishing write tid 3 to trial03942481-39 2026-02-25T08:45:39.467 INFO:tasks.rados.rados.0.trial039.stdout:931: finishing write tid 4 to trial03942481-39 2026-02-25T08:45:39.470 INFO:tasks.rados.rados.0.trial039.stdout:931: finishing write tid 5 to trial03942481-39 2026-02-25T08:45:39.470 INFO:tasks.rados.rados.0.trial039.stdout:931: oid 39 updating version 0 to 336 2026-02-25T08:45:39.470 INFO:tasks.rados.rados.0.trial039.stdout:931: oid 39 version 336 is already newer than 331 2026-02-25T08:45:39.470 INFO:tasks.rados.rados.0.trial039.stdout:931: oid 39 version 336 is already newer than 336 2026-02-25T08:45:39.470 INFO:tasks.rados.rados.0.trial039.stdout:931: oid 39 version 336 is already newer than 336 2026-02-25T08:45:39.470 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 39 v 336 (ObjNum 445 snap 119 seq_num 445) dirty exists 2026-02-25T08:45:39.471 INFO:tasks.rados.rados.0.trial039.stdout:931: left oid 39 (ObjNum 445 snap 119 seq_num 445) 2026-02-25T08:45:39.478 INFO:tasks.rados.rados.0.trial039.stdout:929: done (2 left) 2026-02-25T08:45:39.478 INFO:tasks.rados.rados.0.trial039.stdout:931: done (1 left) 2026-02-25T08:45:39.478 INFO:tasks.rados.rados.0.trial039.stdout:932: done (0 left) 2026-02-25T08:45:39.478 INFO:tasks.rados.rados.0.trial039.stdout:933: read oid 38 snap -1 2026-02-25T08:45:39.479 INFO:tasks.rados.rados.0.trial039.stdout:933: expect (ObjNum 408 snap 111 seq_num 408) 2026-02-25T08:45:39.479 INFO:tasks.rados.rados.0.trial039.stdout:934: write oid 13 current snap is 119 2026-02-25T08:45:39.479 INFO:tasks.rados.rados.0.trial039.stdout:934: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2026-02-25T08:45:39.484 INFO:tasks.rados.rados.0.trial039.stdout:934: writing trial03942481-13 from 712729 to 1497506 tid 1 2026-02-25T08:45:39.488 INFO:tasks.rados.rados.0.trial039.stdout:934: writing trial03942481-13 from 2091098 to 2558589 tid 2 2026-02-25T08:45:39.492 INFO:tasks.rados.rados.0.trial039.stdout:934: writing trial03942481-13 from 3342313 to 3512729 tid 3 2026-02-25T08:45:39.492 INFO:tasks.rados.rados.0.trial039.stdout:935: write oid 17 current snap is 119 2026-02-25T08:45:39.493 INFO:tasks.rados.rados.0.trial039.stdout:935: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2026-02-25T08:45:39.497 INFO:tasks.rados.rados.0.trial039.stdout:935: writing trial03942481-17 from 729536 to 1189562 tid 1 2026-02-25T08:45:39.501 INFO:tasks.rados.rados.0.trial039.stdout:935: writing trial03942481-17 from 1749580 to 2360729 tid 2 2026-02-25T08:45:39.505 INFO:tasks.rados.rados.0.trial039.stdout:935: writing trial03942481-17 from 2853383 to 3305833 tid 3 2026-02-25T08:45:39.506 INFO:tasks.rados.rados.0.trial039.stdout:936: read oid 14 snap -1 2026-02-25T08:45:39.506 INFO:tasks.rados.rados.0.trial039.stdout:936: expect (ObjNum 365 snap 97 seq_num 365) 2026-02-25T08:45:39.506 INFO:tasks.rados.rados.0.trial039.stdout:937: write oid 42 current snap is 119 2026-02-25T08:45:39.506 INFO:tasks.rados.rados.0.trial039.stdout:937: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2026-02-25T08:45:39.510 INFO:tasks.rados.rados.0.trial039.stdout:937: writing trial03942481-42 from 746343 to 1397971 tid 1 2026-02-25T08:45:39.515 INFO:tasks.rados.rados.0.trial039.stdout:937: writing trial03942481-42 from 1924415 to 2679222 tid 2 2026-02-25T08:45:39.519 INFO:tasks.rados.rados.0.trial039.stdout:937: writing trial03942481-42 from 3397159 to 3546343 tid 3 2026-02-25T08:45:39.519 INFO:tasks.rados.rados.0.trial039.stdout:938: snap_remove snap 118 2026-02-25T08:45:40.181 INFO:tasks.rados.rados.0.trial039.stdout:938: done (5 left) 2026-02-25T08:45:40.181 INFO:tasks.rados.rados.0.trial039.stdout:939: snap_remove snap 117 2026-02-25T08:45:40.758 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:40.834 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:40.830+0000 7fe991d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:40.837 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:40.834+0000 7fe991d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:40.865 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:40.882 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:41.184 INFO:tasks.rados.rados.0.trial039.stdout:939: done (5 left) 2026-02-25T08:45:41.184 INFO:tasks.rados.rados.0.trial039.stdout:940: snap_remove snap 111 2026-02-25T08:45:42.187 INFO:tasks.rados.rados.0.trial039.stdout:940: done (5 left) 2026-02-25T08:45:42.188 INFO:tasks.rados.rados.0.trial039.stdout:941: write (excl) oid 34 current snap is 119 2026-02-25T08:45:42.188 INFO:tasks.rados.rados.0.trial039.stdout:933: expect (ObjNum 408 snap 111 seq_num 408) 2026-02-25T08:45:42.189 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 949312 2026-02-25T08:45:42.226 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 904982 2026-02-25T08:45:42.243 INFO:tasks.rados.rados.0.trial039.stdout:941: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2026-02-25T08:45:42.249 INFO:tasks.rados.rados.0.trial039.stdout:941: writing trial03942481-34 from 763150 to 1490027 tid 1 2026-02-25T08:45:42.253 INFO:tasks.rados.rados.0.trial039.stdout:941: writing trial03942481-34 from 1982897 to 2597715 tid 2 2026-02-25T08:45:42.257 INFO:tasks.rados.rados.0.trial039.stdout:941: writing trial03942481-34 from 3024582 to 3499576 tid 3 2026-02-25T08:45:42.257 INFO:tasks.rados.rados.0.trial039.stdout:933: done (5 left) 2026-02-25T08:45:42.258 INFO:tasks.rados.rados.0.trial039.stdout:942: read oid 48 snap -1 2026-02-25T08:45:42.258 INFO:tasks.rados.rados.0.trial039.stdout:942: expect deleted 2026-02-25T08:45:42.258 INFO:tasks.rados.rados.0.trial039.stdout:943: read oid 12 snap -1 2026-02-25T08:45:42.258 INFO:tasks.rados.rados.0.trial039.stdout:943: expect deleted 2026-02-25T08:45:42.258 INFO:tasks.rados.rados.0.trial039.stdout:944: delete oid 4 current snap is 119 2026-02-25T08:45:42.258 INFO:tasks.rados.rados.0.trial039.stdout:935: finishing write tid 1 to trial03942481-17 2026-02-25T08:45:42.258 INFO:tasks.rados.rados.0.trial039.stdout:936: expect (ObjNum 365 snap 97 seq_num 365) 2026-02-25T08:45:42.258 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:42.311 INFO:tasks.rados.rados.0.trial039.stdout:934: finishing write tid 1 to trial03942481-13 2026-02-25T08:45:42.311 INFO:tasks.rados.rados.0.trial039.stdout:935: finishing write tid 2 to trial03942481-17 2026-02-25T08:45:42.311 INFO:tasks.rados.rados.0.trial039.stdout:936: done (7 left) 2026-02-25T08:45:42.312 INFO:tasks.rados.rados.0.trial039.stdout:944: done (6 left) 2026-02-25T08:45:42.312 INFO:tasks.rados.rados.0.trial039.stdout:945: write (excl) oid 40 current snap is 119 2026-02-25T08:45:42.312 INFO:tasks.rados.rados.0.trial039.stdout:934: finishing write tid 2 to trial03942481-13 2026-02-25T08:45:42.312 INFO:tasks.rados.rados.0.trial039.stdout:935: finishing write tid 3 to trial03942481-17 2026-02-25T08:45:42.312 INFO:tasks.rados.rados.0.trial039.stdout:945: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2026-02-25T08:45:42.315 INFO:tasks.rados.rados.0.trial039.stdout:945: writing trial03942481-40 from 796764 to 1274139 tid 1 2026-02-25T08:45:42.320 INFO:tasks.rados.rados.0.trial039.stdout:945: writing trial03942481-40 from 1983508 to 2601995 tid 2 2026-02-25T08:45:42.325 INFO:tasks.rados.rados.0.trial039.stdout:945: writing trial03942481-40 from 3363075 to 3596764 tid 3 2026-02-25T08:45:42.325 INFO:tasks.rados.rados.0.trial039.stdout:946: read oid 26 snap 114 2026-02-25T08:45:42.325 INFO:tasks.rados.rados.0.trial039.stdout:946: expect (ObjNum 403 snap 111 seq_num 403) 2026-02-25T08:45:42.326 INFO:tasks.rados.rados.0.trial039.stdout:947: read oid 30 snap -1 2026-02-25T08:45:42.326 INFO:tasks.rados.rados.0.trial039.stdout:947: expect (ObjNum 404 snap 111 seq_num 404) 2026-02-25T08:45:42.326 INFO:tasks.rados.rados.0.trial039.stdout:948: delete oid 10 current snap is 119 2026-02-25T08:45:42.326 INFO:tasks.rados.rados.0.trial039.stdout:934: finishing write tid 3 to trial03942481-13 2026-02-25T08:45:42.326 INFO:tasks.rados.rados.0.trial039.stdout:937: finishing write tid 1 to trial03942481-42 2026-02-25T08:45:42.326 INFO:tasks.rados.rados.0.trial039.stdout:935: finishing write tid 4 to trial03942481-17 2026-02-25T08:45:42.326 INFO:tasks.rados.rados.0.trial039.stdout:934: finishing write tid 4 to trial03942481-13 2026-02-25T08:45:42.326 INFO:tasks.rados.rados.0.trial039.stdout:934: finishing write tid 5 to trial03942481-13 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:934: oid 13 updating version 0 to 172 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:934: oid 13 version 172 is already newer than 172 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:934: oid 13 version 172 is already newer than 172 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:934: oid 13 version 172 is already newer than 172 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 13 v 172 (ObjNum 447 snap 119 seq_num 447) dirty exists 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:934: left oid 13 (ObjNum 447 snap 119 seq_num 447) 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:935: finishing write tid 5 to trial03942481-17 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:935: oid 17 updating version 0 to 185 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:935: oid 17 version 185 is already newer than 185 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:935: oid 17 version 185 is already newer than 185 2026-02-25T08:45:42.327 INFO:tasks.rados.rados.0.trial039.stdout:935: oid 17 version 185 is already newer than 185 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 17 v 185 (ObjNum 448 snap 119 seq_num 448) dirty exists 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:935: left oid 17 (ObjNum 448 snap 119 seq_num 448) 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:937: finishing write tid 2 to trial03942481-42 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:937: finishing write tid 3 to trial03942481-42 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:937: finishing write tid 4 to trial03942481-42 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:937: finishing write tid 5 to trial03942481-42 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:937: oid 42 updating version 0 to 343 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:937: oid 42 version 343 is already newer than 343 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:937: oid 42 version 343 is already newer than 343 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:937: oid 42 version 343 is already newer than 343 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 42 v 343 (ObjNum 449 snap 119 seq_num 449) dirty exists 2026-02-25T08:45:42.328 INFO:tasks.rados.rados.0.trial039.stdout:937: left oid 42 (ObjNum 449 snap 119 seq_num 449) 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:941: finishing write tid 1 to trial03942481-34 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:941: finishing write tid 2 to trial03942481-34 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:941: finishing write tid 3 to trial03942481-34 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:941: finishing write tid 4 to trial03942481-34 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:941: finishing write tid 5 to trial03942481-34 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:941: oid 34 updating version 0 to 191 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:941: oid 34 version 191 is already newer than 191 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:941: oid 34 version 191 is already newer than 191 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:941: oid 34 version 191 is already newer than 188 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 34 v 191 (ObjNum 450 snap 119 seq_num 450) dirty exists 2026-02-25T08:45:42.329 INFO:tasks.rados.rados.0.trial039.stdout:941: left oid 34 (ObjNum 450 snap 119 seq_num 450) 2026-02-25T08:45:42.361 INFO:tasks.rados.rados.0.trial039.stdout:934: done (9 left) 2026-02-25T08:45:42.361 INFO:tasks.rados.rados.0.trial039.stdout:935: done (8 left) 2026-02-25T08:45:42.361 INFO:tasks.rados.rados.0.trial039.stdout:937: done (7 left) 2026-02-25T08:45:42.361 INFO:tasks.rados.rados.0.trial039.stdout:941: done (6 left) 2026-02-25T08:45:42.361 INFO:tasks.rados.rados.0.trial039.stdout:942: done (5 left) 2026-02-25T08:45:42.361 INFO:tasks.rados.rados.0.trial039.stdout:943: done (4 left) 2026-02-25T08:45:42.362 INFO:tasks.rados.rados.0.trial039.stdout:948: done (3 left) 2026-02-25T08:45:42.362 INFO:tasks.rados.rados.0.trial039.stdout:949: delete oid 44 current snap is 119 2026-02-25T08:45:42.372 INFO:tasks.rados.rados.0.trial039.stdout:945: finishing write tid 1 to trial03942481-40 2026-02-25T08:45:42.381 INFO:tasks.rados.rados.0.trial039.stdout:947: expect (ObjNum 404 snap 111 seq_num 404) 2026-02-25T08:45:42.390 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1267668 2026-02-25T08:45:42.482 INFO:tasks.rados.rados.0.trial039.stdout:947: done (3 left) 2026-02-25T08:45:42.482 INFO:tasks.rados.rados.0.trial039.stdout:949: done (2 left) 2026-02-25T08:45:42.482 INFO:tasks.rados.rados.0.trial039.stdout:950: read oid 49 snap -1 2026-02-25T08:45:42.482 INFO:tasks.rados.rados.0.trial039.stdout:950: expect deleted 2026-02-25T08:45:42.482 INFO:tasks.rados.rados.0.trial039.stdout:951: write oid 43 current snap is 119 2026-02-25T08:45:42.483 INFO:tasks.rados.rados.0.trial039.stdout:946: expect (ObjNum 403 snap 111 seq_num 403) 2026-02-25T08:45:42.483 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:42.513 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 365224 2026-02-25T08:45:42.514 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:42.571 INFO:tasks.rados.rados.0.trial039.stdout:945: finishing write tid 2 to trial03942481-40 2026-02-25T08:45:42.571 INFO:tasks.rados.rados.0.trial039.stdout:951: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2026-02-25T08:45:42.575 INFO:tasks.rados.rados.0.trial039.stdout:951: writing trial03942481-43 from 447185 to 1150307 tid 1 2026-02-25T08:45:42.580 INFO:tasks.rados.rados.0.trial039.stdout:951: writing trial03942481-43 from 1642601 to 2408415 tid 2 2026-02-25T08:45:42.584 INFO:tasks.rados.rados.0.trial039.stdout:951: writing trial03942481-43 from 2928991 to 3376698 tid 3 2026-02-25T08:45:42.584 INFO:tasks.rados.rados.0.trial039.stdout:946: done (3 left) 2026-02-25T08:45:42.584 INFO:tasks.rados.rados.0.trial039.stdout:952: read oid 12 snap 116 2026-02-25T08:45:42.584 INFO:tasks.rados.rados.0.trial039.stdout:952: expect deleted 2026-02-25T08:45:42.584 INFO:tasks.rados.rados.0.trial039.stdout:953: snap_remove snap 112 2026-02-25T08:45:42.586 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:42.590 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:42.883 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:42.960 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:42.956+0000 7f74bfff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:42.962 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:42.960+0000 7f74bfff5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:42.990 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:43.006 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:43.007 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:43.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~248s 2026-02-25T08:45:43.088 INFO:tasks.rados.rados.0.trial039.stdout:953: done (4 left) 2026-02-25T08:45:43.089 INFO:tasks.rados.rados.0.trial039.stdout:954: snap_create 2026-02-25T08:45:43.089 INFO:tasks.rados.rados.0.trial039.stdout:945: finishing write tid 3 to trial03942481-40 2026-02-25T08:45:43.089 INFO:tasks.rados.rados.0.trial039.stdout:945: finishing write tid 4 to trial03942481-40 2026-02-25T08:45:43.089 INFO:tasks.rados.rados.0.trial039.stdout:945: finishing write tid 5 to trial03942481-40 2026-02-25T08:45:43.089 INFO:tasks.rados.rados.0.trial039.stdout:945: oid 40 updating version 0 to 196 2026-02-25T08:45:43.089 INFO:tasks.rados.rados.0.trial039.stdout:945: oid 40 version 196 is already newer than 196 2026-02-25T08:45:43.089 INFO:tasks.rados.rados.0.trial039.stdout:945: oid 40 version 196 is already newer than 196 2026-02-25T08:45:43.089 INFO:tasks.rados.rados.0.trial039.stdout:945: oid 40 version 196 is already newer than 193 2026-02-25T08:45:43.089 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 40 v 196 (ObjNum 452 snap 119 seq_num 452) dirty exists 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:945: left oid 40 (ObjNum 452 snap 119 seq_num 452) 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:951: finishing write tid 1 to trial03942481-43 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:951: finishing write tid 2 to trial03942481-43 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:951: finishing write tid 3 to trial03942481-43 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:951: finishing write tid 4 to trial03942481-43 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:951: finishing write tid 5 to trial03942481-43 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:951: oid 43 updating version 0 to 143 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:951: oid 43 version 143 is already newer than 143 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:951: oid 43 version 143 is already newer than 143 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:951: oid 43 version 143 is already newer than 143 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 43 v 143 (ObjNum 455 snap 119 seq_num 455) dirty exists 2026-02-25T08:45:43.090 INFO:tasks.rados.rados.0.trial039.stdout:951: left oid 43 (ObjNum 455 snap 119 seq_num 455) 2026-02-25T08:45:44.090 INFO:tasks.rados.rados.0.trial039.stdout:945: done (4 left) 2026-02-25T08:45:44.091 INFO:tasks.rados.rados.0.trial039.stdout:950: done (3 left) 2026-02-25T08:45:44.091 INFO:tasks.rados.rados.0.trial039.stdout:951: done (2 left) 2026-02-25T08:45:44.091 INFO:tasks.rados.rados.0.trial039.stdout:952: done (1 left) 2026-02-25T08:45:44.091 INFO:tasks.rados.rados.0.trial039.stdout:954: done (0 left) 2026-02-25T08:45:44.091 INFO:tasks.rados.rados.0.trial039.stdout:955: delete oid 49 current snap is 120 2026-02-25T08:45:44.106 INFO:tasks.rados.rados.0.trial039.stdout:955: done (0 left) 2026-02-25T08:45:44.106 INFO:tasks.rados.rados.0.trial039.stdout:956: read oid 48 snap -1 2026-02-25T08:45:44.106 INFO:tasks.rados.rados.0.trial039.stdout:956: expect deleted 2026-02-25T08:45:44.106 INFO:tasks.rados.rados.0.trial039.stdout:957: delete oid 29 current snap is 120 2026-02-25T08:45:44.143 INFO:tasks.rados.rados.0.trial039.stdout:956: done (1 left) 2026-02-25T08:45:44.144 INFO:tasks.rados.rados.0.trial039.stdout:957: done (0 left) 2026-02-25T08:45:44.144 INFO:tasks.rados.rados.0.trial039.stdout:958: delete oid 48 current snap is 120 2026-02-25T08:45:44.146 INFO:tasks.rados.rados.0.trial039.stdout:958: done (0 left) 2026-02-25T08:45:44.146 INFO:tasks.rados.rados.0.trial039.stdout:959: write oid 46 current snap is 120 2026-02-25T08:45:44.146 INFO:tasks.rados.rados.0.trial039.stdout:959: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2026-02-25T08:45:44.150 INFO:tasks.rados.rados.0.trial039.stdout:959: writing trial03942481-46 from 514413 to 1234884 tid 1 2026-02-25T08:45:44.156 INFO:tasks.rados.rados.0.trial039.stdout:959: writing trial03942481-46 from 1876529 to 2649681 tid 2 2026-02-25T08:45:44.160 INFO:tasks.rados.rados.0.trial039.stdout:959: writing trial03942481-46 from 3438683 to 3714413 tid 3 2026-02-25T08:45:44.160 INFO:tasks.rados.rados.0.trial039.stdout:960: write oid 25 current snap is 120 2026-02-25T08:45:44.160 INFO:tasks.rados.rados.0.trial039.stdout:960: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2026-02-25T08:45:44.165 INFO:tasks.rados.rados.0.trial039.stdout:960: writing trial03942481-25 from 531220 to 1326940 tid 1 2026-02-25T08:45:44.169 INFO:tasks.rados.rados.0.trial039.stdout:960: writing trial03942481-25 from 1935011 to 2451821 tid 2 2026-02-25T08:45:44.173 INFO:tasks.rados.rados.0.trial039.stdout:960: writing trial03942481-25 from 3066106 to 3486526 tid 3 2026-02-25T08:45:44.173 INFO:tasks.rados.rados.0.trial039.stdout:961: write (excl) oid 15 current snap is 120 2026-02-25T08:45:44.173 INFO:tasks.rados.rados.0.trial039.stdout:961: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2026-02-25T08:45:44.177 INFO:tasks.rados.rados.0.trial039.stdout:961: writing trial03942481-15 from 548027 to 1018996 tid 1 2026-02-25T08:45:44.182 INFO:tasks.rados.rados.0.trial039.stdout:961: writing trial03942481-15 from 1593493 to 2370314 tid 2 2026-02-25T08:45:44.187 INFO:tasks.rados.rados.0.trial039.stdout:961: writing trial03942481-15 from 3093529 to 3725221 tid 3 2026-02-25T08:45:44.187 INFO:tasks.rados.rados.0.trial039.stdout:962: snap_remove snap 116 2026-02-25T08:45:45.008 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:45.086 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:45.082+0000 7f4cf619c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:45.089 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:45.087+0000 7f4cf619c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:45.090 INFO:tasks.rados.rados.0.trial039.stdout:962: done (3 left) 2026-02-25T08:45:45.090 INFO:tasks.rados.rados.0.trial039.stdout:963: snap_remove snap 115 2026-02-25T08:45:45.130 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:45.146 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:46.094 INFO:tasks.rados.rados.0.trial039.stdout:963: done (3 left) 2026-02-25T08:45:46.094 INFO:tasks.rados.rados.0.trial039.stdout:964: snap_remove snap 89 2026-02-25T08:45:47.101 INFO:tasks.rados.rados.0.trial039.stdout:964: done (3 left) 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:965: delete oid 24 current snap is 120 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:959: finishing write tid 1 to trial03942481-46 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:959: finishing write tid 2 to trial03942481-46 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:959: finishing write tid 3 to trial03942481-46 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:959: finishing write tid 4 to trial03942481-46 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:959: finishing write tid 5 to trial03942481-46 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:959: oid 46 updating version 0 to 189 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:959: oid 46 version 189 is already newer than 189 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:959: oid 46 version 189 is already newer than 189 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:959: oid 46 version 189 is already newer than 189 2026-02-25T08:45:47.102 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 46 v 189 (ObjNum 459 snap 120 seq_num 459) dirty exists 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:959: left oid 46 (ObjNum 459 snap 120 seq_num 459) 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:961: finishing write tid 1 to trial03942481-15 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:961: finishing write tid 2 to trial03942481-15 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:961: finishing write tid 3 to trial03942481-15 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:961: finishing write tid 4 to trial03942481-15 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:961: finishing write tid 5 to trial03942481-15 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:961: oid 15 updating version 0 to 179 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:961: oid 15 version 179 is already newer than 179 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:961: oid 15 version 179 is already newer than 176 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:961: oid 15 version 179 is already newer than 179 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 15 v 179 (ObjNum 461 snap 120 seq_num 461) dirty exists 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:961: left oid 15 (ObjNum 461 snap 120 seq_num 461) 2026-02-25T08:45:47.103 INFO:tasks.rados.rados.0.trial039.stdout:960: finishing write tid 1 to trial03942481-25 2026-02-25T08:45:47.104 INFO:tasks.rados.rados.0.trial039.stdout:960: finishing write tid 2 to trial03942481-25 2026-02-25T08:45:47.104 INFO:tasks.rados.rados.0.trial039.stdout:960: finishing write tid 3 to trial03942481-25 2026-02-25T08:45:47.104 INFO:tasks.rados.rados.0.trial039.stdout:960: finishing write tid 4 to trial03942481-25 2026-02-25T08:45:47.104 INFO:tasks.rados.rados.0.trial039.stdout:960: finishing write tid 5 to trial03942481-25 2026-02-25T08:45:47.104 INFO:tasks.rados.rados.0.trial039.stdout:960: oid 25 updating version 0 to 56 2026-02-25T08:45:47.104 INFO:tasks.rados.rados.0.trial039.stdout:960: oid 25 version 56 is already newer than 56 2026-02-25T08:45:47.104 INFO:tasks.rados.rados.0.trial039.stdout:960: oid 25 version 56 is already newer than 56 2026-02-25T08:45:47.104 INFO:tasks.rados.rados.0.trial039.stdout:960: oid 25 version 56 is already newer than 56 2026-02-25T08:45:47.104 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 25 v 56 (ObjNum 460 snap 120 seq_num 460) dirty exists 2026-02-25T08:45:47.104 INFO:tasks.rados.rados.0.trial039.stdout:960: left oid 25 (ObjNum 460 snap 120 seq_num 460) 2026-02-25T08:45:47.139 INFO:tasks.rados.rados.0.trial039.stdout:959: done (3 left) 2026-02-25T08:45:47.139 INFO:tasks.rados.rados.0.trial039.stdout:960: done (2 left) 2026-02-25T08:45:47.139 INFO:tasks.rados.rados.0.trial039.stdout:961: done (1 left) 2026-02-25T08:45:47.139 INFO:tasks.rados.rados.0.trial039.stdout:965: done (0 left) 2026-02-25T08:45:47.139 INFO:tasks.rados.rados.0.trial039.stdout:966: read oid 29 snap 119 2026-02-25T08:45:47.139 INFO:tasks.rados.rados.0.trial039.stdout:966: expect (ObjNum 422 snap 115 seq_num 422) 2026-02-25T08:45:47.140 INFO:tasks.rados.rados.0.trial039.stdout:967: snap_remove snap 108 2026-02-25T08:45:47.147 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:47.225 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:47.222+0000 7fb3bfc51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:47.227 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:47.226+0000 7fb3bfc51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:47.254 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:47.270 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:47.591 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:47.662 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:47.666 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:48.100 INFO:tasks.rados.rados.0.trial039.stdout:967: done (1 left) 2026-02-25T08:45:48.100 INFO:tasks.rados.rados.0.trial039.stdout:968: delete oid 44 current snap is 120 2026-02-25T08:45:48.100 INFO:tasks.rados.rados.0.trial039.stdout:966: expect (ObjNum 422 snap 115 seq_num 422) 2026-02-25T08:45:48.161 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:48.239 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1499921 2026-02-25T08:45:48.276 INFO:tasks.rados.rados.0.trial039.stdout:966: done (1 left) 2026-02-25T08:45:48.276 INFO:tasks.rados.rados.0.trial039.stdout:968: done (0 left) 2026-02-25T08:45:48.276 INFO:tasks.rados.rados.0.trial039.stdout:969: write oid 9 current snap is 120 2026-02-25T08:45:48.276 INFO:tasks.rados.rados.0.trial039.stdout:969: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2026-02-25T08:45:48.280 INFO:tasks.rados.rados.0.trial039.stdout:969: writing trial03942481-9 from 598448 to 1011517 tid 1 2026-02-25T08:45:48.285 INFO:tasks.rados.rados.0.trial039.stdout:969: writing trial03942481-9 from 1768939 to 2293087 tid 2 2026-02-25T08:45:48.289 INFO:tasks.rados.rados.0.trial039.stdout:969: writing trial03942481-9 from 2775798 to 3357659 tid 3 2026-02-25T08:45:48.291 INFO:tasks.rados.rados.0.trial039.stdout:969: writing trial03942481-9 from 3781488 to 3798448 tid 4 2026-02-25T08:45:48.291 INFO:tasks.rados.rados.0.trial039.stdout:970: snap_remove snap 114 2026-02-25T08:45:48.514 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~253s 2026-02-25T08:45:49.106 INFO:tasks.rados.rados.0.trial039.stdout:970: done (1 left) 2026-02-25T08:45:49.106 INFO:tasks.rados.rados.0.trial039.stdout:971: write (excl) oid 15 current snap is 120 2026-02-25T08:45:49.106 INFO:tasks.rados.rados.0.trial039.stdout:971: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2026-02-25T08:45:49.110 INFO:tasks.rados.rados.0.trial039.stdout:971: writing trial03942481-15 from 615255 to 1103573 tid 1 2026-02-25T08:45:49.117 INFO:tasks.rados.rados.0.trial039.stdout:971: writing trial03942481-15 from 1827421 to 2611580 tid 2 2026-02-25T08:45:49.122 INFO:tasks.rados.rados.0.trial039.stdout:971: writing trial03942481-15 from 3203221 to 3815255 tid 3 2026-02-25T08:45:49.122 INFO:tasks.rados.rados.0.trial039.stdout:972: delete oid 4 current snap is 120 2026-02-25T08:45:49.122 INFO:tasks.rados.rados.0.trial039.stdout:969: finishing write tid 1 to trial03942481-9 2026-02-25T08:45:49.122 INFO:tasks.rados.rados.0.trial039.stdout:969: finishing write tid 2 to trial03942481-9 2026-02-25T08:45:49.122 INFO:tasks.rados.rados.0.trial039.stdout:969: finishing write tid 3 to trial03942481-9 2026-02-25T08:45:49.122 INFO:tasks.rados.rados.0.trial039.stdout:969: finishing write tid 4 to trial03942481-9 2026-02-25T08:45:49.122 INFO:tasks.rados.rados.0.trial039.stdout:969: finishing write tid 5 to trial03942481-9 2026-02-25T08:45:49.122 INFO:tasks.rados.rados.0.trial039.stdout:969: finishing write tid 6 to trial03942481-9 2026-02-25T08:45:49.122 INFO:tasks.rados.rados.0.trial039.stdout:969: oid 9 updating version 0 to 44 2026-02-25T08:45:49.123 INFO:tasks.rados.rados.0.trial039.stdout:969: oid 9 version 44 is already newer than 44 2026-02-25T08:45:49.123 INFO:tasks.rados.rados.0.trial039.stdout:969: oid 9 version 44 is already newer than 44 2026-02-25T08:45:49.123 INFO:tasks.rados.rados.0.trial039.stdout:969: oid 9 version 44 is already newer than 44 2026-02-25T08:45:49.123 INFO:tasks.rados.rados.0.trial039.stdout:969: oid 9 version 44 is already newer than 44 2026-02-25T08:45:49.123 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 9 v 44 (ObjNum 464 snap 120 seq_num 464) dirty exists 2026-02-25T08:45:49.123 INFO:tasks.rados.rados.0.trial039.stdout:969: left oid 9 (ObjNum 464 snap 120 seq_num 464) 2026-02-25T08:45:49.137 INFO:tasks.rados.rados.0.trial039.stdout:971: finishing write tid 1 to trial03942481-15 2026-02-25T08:45:49.146 INFO:tasks.rados.rados.0.trial039.stdout:969: done (2 left) 2026-02-25T08:45:49.147 INFO:tasks.rados.rados.0.trial039.stdout:972: done (1 left) 2026-02-25T08:45:49.147 INFO:tasks.rados.rados.0.trial039.stdout:973: delete oid 49 current snap is 120 2026-02-25T08:45:49.149 INFO:tasks.rados.rados.0.trial039.stdout:973: done (1 left) 2026-02-25T08:45:49.149 INFO:tasks.rados.rados.0.trial039.stdout:974: write oid 20 current snap is 120 2026-02-25T08:45:49.149 INFO:tasks.rados.rados.0.trial039.stdout:974: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2026-02-25T08:45:49.154 INFO:tasks.rados.rados.0.trial039.stdout:974: writing trial03942481-20 from 665676 to 1379741 tid 1 2026-02-25T08:45:49.158 INFO:tasks.rados.rados.0.trial039.stdout:974: writing trial03942481-20 from 1886514 to 2418000 tid 2 2026-02-25T08:45:49.164 INFO:tasks.rados.rados.0.trial039.stdout:974: writing trial03942481-20 from 3169137 to 3865676 tid 3 2026-02-25T08:45:49.164 INFO:tasks.rados.rados.0.trial039.stdout:975: snap_create 2026-02-25T08:45:49.164 INFO:tasks.rados.rados.0.trial039.stdout:971: finishing write tid 2 to trial03942481-15 2026-02-25T08:45:49.164 INFO:tasks.rados.rados.0.trial039.stdout:971: finishing write tid 3 to trial03942481-15 2026-02-25T08:45:49.169 INFO:tasks.rados.rados.0.trial039.stdout:971: finishing write tid 4 to trial03942481-15 2026-02-25T08:45:49.172 INFO:tasks.rados.rados.0.trial039.stdout:971: finishing write tid 5 to trial03942481-15 2026-02-25T08:45:49.172 INFO:tasks.rados.rados.0.trial039.stdout:971: oid 15 updating version 0 to 183 2026-02-25T08:45:49.172 INFO:tasks.rados.rados.0.trial039.stdout:971: oid 15 version 183 is already newer than 183 2026-02-25T08:45:49.172 INFO:tasks.rados.rados.0.trial039.stdout:971: oid 15 version 183 is already newer than 180 2026-02-25T08:45:49.173 INFO:tasks.rados.rados.0.trial039.stdout:971: oid 15 version 183 is already newer than 183 2026-02-25T08:45:49.173 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 15 v 183 (ObjNum 465 snap 120 seq_num 465) dirty exists 2026-02-25T08:45:49.173 INFO:tasks.rados.rados.0.trial039.stdout:971: left oid 15 (ObjNum 465 snap 120 seq_num 465) 2026-02-25T08:45:49.195 INFO:tasks.rados.rados.0.trial039.stdout:974: finishing write tid 1 to trial03942481-20 2026-02-25T08:45:49.201 INFO:tasks.rados.rados.0.trial039.stdout:974: finishing write tid 2 to trial03942481-20 2026-02-25T08:45:49.210 INFO:tasks.rados.rados.0.trial039.stdout:974: finishing write tid 3 to trial03942481-20 2026-02-25T08:45:49.214 INFO:tasks.rados.rados.0.trial039.stdout:974: finishing write tid 4 to trial03942481-20 2026-02-25T08:45:49.216 INFO:tasks.rados.rados.0.trial039.stdout:974: finishing write tid 5 to trial03942481-20 2026-02-25T08:45:49.216 INFO:tasks.rados.rados.0.trial039.stdout:974: oid 20 updating version 0 to 201 2026-02-25T08:45:49.216 INFO:tasks.rados.rados.0.trial039.stdout:974: oid 20 version 201 is already newer than 201 2026-02-25T08:45:49.216 INFO:tasks.rados.rados.0.trial039.stdout:974: oid 20 version 201 is already newer than 201 2026-02-25T08:45:49.216 INFO:tasks.rados.rados.0.trial039.stdout:974: oid 20 version 201 is already newer than 201 2026-02-25T08:45:49.216 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 20 v 201 (ObjNum 468 snap 120 seq_num 468) dirty exists 2026-02-25T08:45:49.216 INFO:tasks.rados.rados.0.trial039.stdout:974: left oid 20 (ObjNum 468 snap 120 seq_num 468) 2026-02-25T08:45:49.271 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:49.346 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:49.344+0000 7f96a30cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:49.349 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:49.347+0000 7f96a30cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:49.375 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:49.391 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:50.108 INFO:tasks.rados.rados.0.trial039.stdout:971: done (2 left) 2026-02-25T08:45:50.108 INFO:tasks.rados.rados.0.trial039.stdout:974: done (1 left) 2026-02-25T08:45:50.108 INFO:tasks.rados.rados.0.trial039.stdout:975: done (0 left) 2026-02-25T08:45:50.108 INFO:tasks.rados.rados.0.trial039.stdout:976: delete oid 16 current snap is 121 2026-02-25T08:45:50.124 INFO:tasks.rados.rados.0.trial039.stdout:976: done (0 left) 2026-02-25T08:45:50.124 INFO:tasks.rados.rados.0.trial039.stdout:977: snap_create 2026-02-25T08:45:51.112 INFO:tasks.rados.rados.0.trial039.stdout:977: done (0 left) 2026-02-25T08:45:51.112 INFO:tasks.rados.rados.0.trial039.stdout:978: write (excl) oid 50 current snap is 122 2026-02-25T08:45:51.113 INFO:tasks.rados.rados.0.trial039.stdout:978: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2026-02-25T08:45:51.117 INFO:tasks.rados.rados.0.trial039.stdout:978: writing trial03942481-50 from 699290 to 1163853 tid 1 2026-02-25T08:45:51.121 INFO:tasks.rados.rados.0.trial039.stdout:978: writing trial03942481-50 from 1603478 to 2138633 tid 2 2026-02-25T08:45:51.126 INFO:tasks.rados.rados.0.trial039.stdout:978: writing trial03942481-50 from 2823983 to 3589829 tid 3 2026-02-25T08:45:51.127 INFO:tasks.rados.rados.0.trial039.stdout:979: read oid 37 snap 120 2026-02-25T08:45:51.127 INFO:tasks.rados.rados.0.trial039.stdout:979: expect deleted 2026-02-25T08:45:51.127 INFO:tasks.rados.rados.0.trial039.stdout:980: delete oid 41 current snap is 122 2026-02-25T08:45:51.178 INFO:tasks.rados.rados.0.trial039.stdout:978: finishing write tid 1 to trial03942481-50 2026-02-25T08:45:51.198 INFO:tasks.rados.rados.0.trial039.stdout:978: finishing write tid 2 to trial03942481-50 2026-02-25T08:45:51.211 INFO:tasks.rados.rados.0.trial039.stdout:978: finishing write tid 3 to trial03942481-50 2026-02-25T08:45:51.224 INFO:tasks.rados.rados.0.trial039.stdout:978: finishing write tid 4 to trial03942481-50 2026-02-25T08:45:51.228 INFO:tasks.rados.rados.0.trial039.stdout:978: finishing write tid 5 to trial03942481-50 2026-02-25T08:45:51.228 INFO:tasks.rados.rados.0.trial039.stdout:978: oid 50 updating version 0 to 203 2026-02-25T08:45:51.228 INFO:tasks.rados.rados.0.trial039.stdout:978: oid 50 updating version 203 to 206 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:978: oid 50 version 206 is already newer than 206 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:978: oid 50 version 206 is already newer than 206 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 50 v 206 (ObjNum 470 snap 122 seq_num 470) dirty exists 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:978: left oid 50 (ObjNum 470 snap 122 seq_num 470) 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:978: done (2 left) 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:979: done (1 left) 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:980: done (0 left) 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:981: read oid 4 snap 120 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:981: expect deleted 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:982: write (excl) oid 17 current snap is 122 2026-02-25T08:45:51.229 INFO:tasks.rados.rados.0.trial039.stdout:982: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2026-02-25T08:45:51.234 INFO:tasks.rados.rados.0.trial039.stdout:982: writing trial03942481-17 from 732904 to 1464318 tid 1 2026-02-25T08:45:51.238 INFO:tasks.rados.rados.0.trial039.stdout:982: writing trial03942481-17 from 2120442 to 2659266 tid 2 2026-02-25T08:45:51.243 INFO:tasks.rados.rados.0.trial039.stdout:982: writing trial03942481-17 from 3278829 to 3783572 tid 3 2026-02-25T08:45:51.243 INFO:tasks.rados.rados.0.trial039.stdout:983: read oid 9 snap -1 2026-02-25T08:45:51.243 INFO:tasks.rados.rados.0.trial039.stdout:983: expect (ObjNum 464 snap 120 seq_num 464) 2026-02-25T08:45:51.243 INFO:tasks.rados.rados.0.trial039.stdout:984: write oid 33 current snap is 122 2026-02-25T08:45:51.243 INFO:tasks.rados.rados.0.trial039.stdout:984: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2026-02-25T08:45:51.247 INFO:tasks.rados.rados.0.trial039.stdout:984: writing trial03942481-33 from 749711 to 1156374 tid 1 2026-02-25T08:45:51.252 INFO:tasks.rados.rados.0.trial039.stdout:984: writing trial03942481-33 from 1778924 to 2461406 tid 2 2026-02-25T08:45:51.257 INFO:tasks.rados.rados.0.trial039.stdout:984: writing trial03942481-33 from 3189899 to 3905914 tid 3 2026-02-25T08:45:51.258 INFO:tasks.rados.rados.0.trial039.stdout:985: delete oid 32 current snap is 122 2026-02-25T08:45:51.277 INFO:tasks.rados.rados.0.trial039.stdout:982: finishing write tid 1 to trial03942481-17 2026-02-25T08:45:51.298 INFO:tasks.rados.rados.0.trial039.stdout:982: finishing write tid 2 to trial03942481-17 2026-02-25T08:45:51.298 INFO:tasks.rados.rados.0.trial039.stdout:984: finishing write tid 1 to trial03942481-33 2026-02-25T08:45:51.314 INFO:tasks.rados.rados.0.trial039.stdout:984: finishing write tid 2 to trial03942481-33 2026-02-25T08:45:51.318 INFO:tasks.rados.rados.0.trial039.stdout:982: finishing write tid 3 to trial03942481-17 2026-02-25T08:45:51.323 INFO:tasks.rados.rados.0.trial039.stdout:982: finishing write tid 4 to trial03942481-17 2026-02-25T08:45:51.327 INFO:tasks.rados.rados.0.trial039.stdout:982: finishing write tid 5 to trial03942481-17 2026-02-25T08:45:51.327 INFO:tasks.rados.rados.0.trial039.stdout:982: oid 17 updating version 0 to 192 2026-02-25T08:45:51.328 INFO:tasks.rados.rados.0.trial039.stdout:982: oid 17 version 192 is already newer than 192 2026-02-25T08:45:51.328 INFO:tasks.rados.rados.0.trial039.stdout:982: oid 17 version 192 is already newer than 189 2026-02-25T08:45:51.328 INFO:tasks.rados.rados.0.trial039.stdout:982: oid 17 version 192 is already newer than 192 2026-02-25T08:45:51.328 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 17 v 192 (ObjNum 472 snap 122 seq_num 472) dirty exists 2026-02-25T08:45:51.328 INFO:tasks.rados.rados.0.trial039.stdout:982: left oid 17 (ObjNum 472 snap 122 seq_num 472) 2026-02-25T08:45:51.331 INFO:tasks.rados.rados.0.trial039.stdout:981: done (4 left) 2026-02-25T08:45:51.331 INFO:tasks.rados.rados.0.trial039.stdout:982: done (3 left) 2026-02-25T08:45:51.331 INFO:tasks.rados.rados.0.trial039.stdout:985: done (2 left) 2026-02-25T08:45:51.331 INFO:tasks.rados.rados.0.trial039.stdout:986: snap_create 2026-02-25T08:45:51.332 INFO:tasks.rados.rados.0.trial039.stdout:984: finishing write tid 3 to trial03942481-33 2026-02-25T08:45:51.333 INFO:tasks.rados.rados.0.trial039.stdout:983: expect (ObjNum 464 snap 120 seq_num 464) 2026-02-25T08:45:51.333 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:51.392 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:51.469 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:51.467+0000 7f5b0d5e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:51.471 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:51.470+0000 7f5b0d5e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:51.472 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:51.498 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:51.514 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:51.515 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:51.540 INFO:tasks.rados.rados.0.trial039.stdout:984: finishing write tid 4 to trial03942481-33 2026-02-25T08:45:51.541 INFO:tasks.rados.rados.0.trial039.stdout:984: finishing write tid 5 to trial03942481-33 2026-02-25T08:45:51.541 INFO:tasks.rados.rados.0.trial039.stdout:984: oid 33 updating version 0 to 148 2026-02-25T08:45:51.541 INFO:tasks.rados.rados.0.trial039.stdout:984: oid 33 version 148 is already newer than 148 2026-02-25T08:45:51.541 INFO:tasks.rados.rados.0.trial039.stdout:984: oid 33 version 148 is already newer than 148 2026-02-25T08:45:51.541 INFO:tasks.rados.rados.0.trial039.stdout:984: oid 33 version 148 is already newer than 148 2026-02-25T08:45:51.541 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 33 v 148 (ObjNum 473 snap 122 seq_num 473) dirty exists 2026-02-25T08:45:51.541 INFO:tasks.rados.rados.0.trial039.stdout:984: left oid 33 (ObjNum 473 snap 122 seq_num 473) 2026-02-25T08:45:52.116 INFO:tasks.rados.rados.0.trial039.stdout:983: done (2 left) 2026-02-25T08:45:52.116 INFO:tasks.rados.rados.0.trial039.stdout:984: done (1 left) 2026-02-25T08:45:52.116 INFO:tasks.rados.rados.0.trial039.stdout:986: done (0 left) 2026-02-25T08:45:52.116 INFO:tasks.rados.rados.0.trial039.stdout:987: snap_remove snap 120 2026-02-25T08:45:52.667 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:52.736 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:52.739 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:52.740 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:53.092 INFO:tasks.rados.rados.0.trial039.stdout:987: done (0 left) 2026-02-25T08:45:53.092 INFO:tasks.rados.rados.0.trial039.stdout:988: snap_create 2026-02-25T08:45:53.516 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:53.594 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:53.591+0000 7f79ee6c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:53.596 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:53.595+0000 7f79ee6c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:53.623 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:53.638 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:53.639 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:54.019 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~259s 2026-02-25T08:45:54.093 INFO:tasks.rados.rados.0.trial039.stdout:988: done (0 left) 2026-02-25T08:45:54.094 INFO:tasks.rados.rados.0.trial039.stdout:989: write (excl) oid 17 current snap is 124 2026-02-25T08:45:54.094 INFO:tasks.rados.rados.0.trial039.stdout:989: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2026-02-25T08:45:54.098 INFO:tasks.rados.rados.0.trial039.stdout:989: writing trial03942481-17 from 783325 to 1340486 tid 1 2026-02-25T08:45:54.103 INFO:tasks.rados.rados.0.trial039.stdout:989: writing trial03942481-17 from 1779535 to 2465686 tid 2 2026-02-25T08:45:54.107 INFO:tasks.rados.rados.0.trial039.stdout:989: writing trial03942481-17 from 3244745 to 3699657 tid 3 2026-02-25T08:45:54.108 INFO:tasks.rados.rados.0.trial039.stdout:990: snap_create 2026-02-25T08:45:54.134 INFO:tasks.rados.rados.0.trial039.stdout:989: finishing write tid 1 to trial03942481-17 2026-02-25T08:45:54.150 INFO:tasks.rados.rados.0.trial039.stdout:989: finishing write tid 2 to trial03942481-17 2026-02-25T08:45:54.159 INFO:tasks.rados.rados.0.trial039.stdout:989: finishing write tid 3 to trial03942481-17 2026-02-25T08:45:54.164 INFO:tasks.rados.rados.0.trial039.stdout:989: finishing write tid 4 to trial03942481-17 2026-02-25T08:45:54.165 INFO:tasks.rados.rados.0.trial039.stdout:989: finishing write tid 5 to trial03942481-17 2026-02-25T08:45:54.166 INFO:tasks.rados.rados.0.trial039.stdout:989: oid 17 updating version 0 to 197 2026-02-25T08:45:54.166 INFO:tasks.rados.rados.0.trial039.stdout:989: oid 17 version 197 is already newer than 194 2026-02-25T08:45:54.166 INFO:tasks.rados.rados.0.trial039.stdout:989: oid 17 version 197 is already newer than 197 2026-02-25T08:45:54.166 INFO:tasks.rados.rados.0.trial039.stdout:989: oid 17 version 197 is already newer than 197 2026-02-25T08:45:54.166 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 17 v 197 (ObjNum 475 snap 124 seq_num 475) dirty exists 2026-02-25T08:45:54.166 INFO:tasks.rados.rados.0.trial039.stdout:989: left oid 17 (ObjNum 475 snap 124 seq_num 475) 2026-02-25T08:45:55.094 INFO:tasks.rados.rados.0.trial039.stdout:989: done (1 left) 2026-02-25T08:45:55.094 INFO:tasks.rados.rados.0.trial039.stdout:990: done (0 left) 2026-02-25T08:45:55.094 INFO:tasks.rados.rados.0.trial039.stdout:991: delete oid 13 current snap is 125 2026-02-25T08:45:55.142 INFO:tasks.rados.rados.0.trial039.stdout:991: done (0 left) 2026-02-25T08:45:55.142 INFO:tasks.rados.rados.0.trial039.stdout:992: snap_create 2026-02-25T08:45:55.640 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:55.718 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:55.715+0000 7fe777774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:55.720 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:55.719+0000 7fe777774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:55.748 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:55.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:55.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:56.123 INFO:tasks.rados.rados.0.trial039.stdout:992: done (0 left) 2026-02-25T08:45:56.123 INFO:tasks.rados.rados.0.trial039.stdout:993: snap_remove snap 119 2026-02-25T08:45:57.126 INFO:tasks.rados.rados.0.trial039.stdout:993: done (0 left) 2026-02-25T08:45:57.126 INFO:tasks.rados.rados.0.trial039.stdout:994: delete oid 1 current snap is 126 2026-02-25T08:45:57.171 INFO:tasks.rados.rados.0.trial039.stdout:994: done (0 left) 2026-02-25T08:45:57.171 INFO:tasks.rados.rados.0.trial039.stdout:995: delete oid 38 current snap is 126 2026-02-25T08:45:57.208 INFO:tasks.rados.rados.0.trial039.stdout:995: done (0 left) 2026-02-25T08:45:57.208 INFO:tasks.rados.rados.0.trial039.stdout:996: write (excl) oid 33 current snap is 126 2026-02-25T08:45:57.209 INFO:tasks.rados.rados.0.trial039.stdout:996: seq_num 479 ranges {450553=574510,1613463=437090} 2026-02-25T08:45:57.212 INFO:tasks.rados.rados.0.trial039.stdout:996: writing trial03942481-33 from 450553 to 1025063 tid 1 2026-02-25T08:45:57.217 INFO:tasks.rados.rados.0.trial039.stdout:996: writing trial03942481-33 from 1613463 to 2050553 tid 2 2026-02-25T08:45:57.217 INFO:tasks.rados.rados.0.trial039.stdout:997: snap_create 2026-02-25T08:45:57.240 INFO:tasks.rados.rados.0.trial039.stdout:996: finishing write tid 1 to trial03942481-33 2026-02-25T08:45:57.255 INFO:tasks.rados.rados.0.trial039.stdout:996: finishing write tid 2 to trial03942481-33 2026-02-25T08:45:57.263 INFO:tasks.rados.rados.0.trial039.stdout:996: finishing write tid 3 to trial03942481-33 2026-02-25T08:45:57.264 INFO:tasks.rados.rados.0.trial039.stdout:996: finishing write tid 4 to trial03942481-33 2026-02-25T08:45:57.265 INFO:tasks.rados.rados.0.trial039.stdout:996: oid 33 updating version 0 to 152 2026-02-25T08:45:57.265 INFO:tasks.rados.rados.0.trial039.stdout:996: oid 33 version 152 is already newer than 152 2026-02-25T08:45:57.265 INFO:tasks.rados.rados.0.trial039.stdout:996: oid 33 version 152 is already newer than 150 2026-02-25T08:45:57.265 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 33 v 152 (ObjNum 479 snap 126 seq_num 479) dirty exists 2026-02-25T08:45:57.265 INFO:tasks.rados.rados.0.trial039.stdout:996: left oid 33 (ObjNum 479 snap 126 seq_num 479) 2026-02-25T08:45:57.740 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:45:57.764 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:57.811 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:57.814 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:45:57.840 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:57.838+0000 7f8709246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:57.842 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:57.841+0000 7f8709246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:57.869 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:45:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:57.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:58.132 INFO:tasks.rados.rados.0.trial039.stdout:996: done (1 left) 2026-02-25T08:45:58.132 INFO:tasks.rados.rados.0.trial039.stdout:997: done (0 left) 2026-02-25T08:45:58.132 INFO:tasks.rados.rados.0.trial039.stdout:998: read oid 17 snap -1 2026-02-25T08:45:58.132 INFO:tasks.rados.rados.0.trial039.stdout:998: expect (ObjNum 475 snap 124 seq_num 475) 2026-02-25T08:45:58.132 INFO:tasks.rados.rados.0.trial039.stdout:999: read oid 9 snap -1 2026-02-25T08:45:58.132 INFO:tasks.rados.rados.0.trial039.stdout:999: expect (ObjNum 464 snap 120 seq_num 464) 2026-02-25T08:45:58.133 INFO:tasks.rados.rados.0.trial039.stdout:1000: write oid 7 current snap is 127 2026-02-25T08:45:58.133 INFO:tasks.rados.rados.0.trial039.stdout:1000: seq_num 480 ranges {467360=649759,1671945=395415} 2026-02-25T08:45:58.135 INFO:tasks.rados.rados.0.trial039.stdout:1000: writing trial03942481-7 from 467360 to 1117119 tid 1 2026-02-25T08:45:58.139 INFO:tasks.rados.rados.0.trial039.stdout:1000: writing trial03942481-7 from 1671945 to 2067360 tid 2 2026-02-25T08:45:58.139 INFO:tasks.rados.rados.0.trial039.stdout:1001: snap_remove snap 124 2026-02-25T08:45:59.136 INFO:tasks.rados.rados.0.trial039.stdout:1001: done (3 left) 2026-02-25T08:45:59.136 INFO:tasks.rados.rados.0.trial039.stdout:1002: delete oid 14 current snap is 127 2026-02-25T08:45:59.136 INFO:tasks.rados.rados.0.trial039.stdout:999: expect (ObjNum 464 snap 120 seq_num 464) 2026-02-25T08:45:59.316 INFO:tasks.rados.rados.0.trial039.stdout:998: expect (ObjNum 475 snap 124 seq_num 475) 2026-02-25T08:45:59.317 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:59.377 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:59.438 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:45:59.499 INFO:tasks.rados.rados.0.trial039.stdout:1000: finishing write tid 1 to trial03942481-7 2026-02-25T08:45:59.499 INFO:tasks.rados.rados.0.trial039.stdout:1000: finishing write tid 2 to trial03942481-7 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:1000: finishing write tid 3 to trial03942481-7 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:1000: finishing write tid 4 to trial03942481-7 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:1000: oid 7 updating version 0 to 201 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:1000: oid 7 version 201 is already newer than 201 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:1000: oid 7 version 201 is already newer than 201 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 7 v 201 (ObjNum 480 snap 127 seq_num 480) dirty exists 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:1000: left oid 7 (ObjNum 480 snap 127 seq_num 480) 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:998: done (3 left) 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:999: done (2 left) 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:1000: done (1 left) 2026-02-25T08:45:59.500 INFO:tasks.rados.rados.0.trial039.stdout:1002: done (0 left) 2026-02-25T08:45:59.501 INFO:tasks.rados.rados.0.trial039.stdout:1003: snap_remove snap 121 2026-02-25T08:45:59.522 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~264s 2026-02-25T08:45:59.886 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:59.962 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:59.959+0000 7f1c19db8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:59.964 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:45:59.963+0000 7f1c19db8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:59.992 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:00.009 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:00.139 INFO:tasks.rados.rados.0.trial039.stdout:1003: done (0 left) 2026-02-25T08:46:00.139 INFO:tasks.rados.rados.0.trial039.stdout:1004: snap_create 2026-02-25T08:46:01.142 INFO:tasks.rados.rados.0.trial039.stdout:1004: done (0 left) 2026-02-25T08:46:01.142 INFO:tasks.rados.rados.0.trial039.stdout:1005: read oid 43 snap -1 2026-02-25T08:46:01.143 INFO:tasks.rados.rados.0.trial039.stdout:1005: expect (ObjNum 455 snap 119 seq_num 455) 2026-02-25T08:46:01.143 INFO:tasks.rados.rados.0.trial039.stdout:1006: snap_remove snap 127 2026-02-25T08:46:02.011 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:02.087 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:02.085+0000 7f16841b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:02.090 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:02.089+0000 7f16841b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:02.118 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:02.134 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:02.145 INFO:tasks.rados.rados.0.trial039.stdout:1006: done (1 left) 2026-02-25T08:46:02.145 INFO:tasks.rados.rados.0.trial039.stdout:1007: read oid 45 snap -1 2026-02-25T08:46:02.146 INFO:tasks.rados.rados.0.trial039.stdout:1007: expect deleted 2026-02-25T08:46:02.146 INFO:tasks.rados.rados.0.trial039.stdout:1008: write oid 8 current snap is 128 2026-02-25T08:46:02.146 INFO:tasks.rados.rados.0.trial039.stdout:1008: seq_num 482 ranges {500974=400257,1388909=557169} 2026-02-25T08:46:02.148 INFO:tasks.rados.rados.0.trial039.stdout:1008: writing trial03942481-8 from 500974 to 901231 tid 1 2026-02-25T08:46:02.152 INFO:tasks.rados.rados.0.trial039.stdout:1008: writing trial03942481-8 from 1388909 to 1946078 tid 2 2026-02-25T08:46:02.153 INFO:tasks.rados.rados.0.trial039.stdout:1009: read oid 20 snap 126 2026-02-25T08:46:02.153 INFO:tasks.rados.rados.0.trial039.stdout:1009: expect (ObjNum 468 snap 120 seq_num 468) 2026-02-25T08:46:02.153 INFO:tasks.rados.rados.0.trial039.stdout:1010: snap_remove snap 123 2026-02-25T08:46:02.814 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:02.887 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:02.890 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:03.094 INFO:tasks.rados.rados.0.trial039.stdout:1010: done (4 left) 2026-02-25T08:46:03.094 INFO:tasks.rados.rados.0.trial039.stdout:1011: snap_remove snap 122 2026-02-25T08:46:04.095 INFO:tasks.rados.rados.0.trial039.stdout:1011: done (4 left) 2026-02-25T08:46:04.095 INFO:tasks.rados.rados.0.trial039.stdout:1012: snap_create 2026-02-25T08:46:04.095 INFO:tasks.rados.rados.0.trial039.stdout:1005: expect (ObjNum 455 snap 119 seq_num 455) 2026-02-25T08:46:04.095 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:04.135 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:04.153 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:04.210 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:04.213 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:04.210+0000 7fe2a1e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:04.216 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:04.214+0000 7fe2a0b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:04.243 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:04.260 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:04.261 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:04.268 INFO:tasks.rados.rados.0.trial039.stdout:1009: expect (ObjNum 468 snap 120 seq_num 468) 2026-02-25T08:46:04.349 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:04.451 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1960149 2026-02-25T08:46:04.500 INFO:tasks.rados.rados.0.trial039.stdout:1008: finishing write tid 1 to trial03942481-8 2026-02-25T08:46:04.500 INFO:tasks.rados.rados.0.trial039.stdout:1008: finishing write tid 2 to trial03942481-8 2026-02-25T08:46:04.500 INFO:tasks.rados.rados.0.trial039.stdout:1008: finishing write tid 3 to trial03942481-8 2026-02-25T08:46:04.500 INFO:tasks.rados.rados.0.trial039.stdout:1008: finishing write tid 4 to trial03942481-8 2026-02-25T08:46:04.500 INFO:tasks.rados.rados.0.trial039.stdout:1008: oid 8 updating version 0 to 156 2026-02-25T08:46:04.500 INFO:tasks.rados.rados.0.trial039.stdout:1008: oid 8 version 156 is already newer than 156 2026-02-25T08:46:04.501 INFO:tasks.rados.rados.0.trial039.stdout:1008: oid 8 version 156 is already newer than 156 2026-02-25T08:46:04.501 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 8 v 156 (ObjNum 482 snap 128 seq_num 482) dirty exists 2026-02-25T08:46:04.501 INFO:tasks.rados.rados.0.trial039.stdout:1008: left oid 8 (ObjNum 482 snap 128 seq_num 482) 2026-02-25T08:46:05.025 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2026-02-25T08:46:05.096 INFO:tasks.rados.rados.0.trial039.stdout:1005: done (4 left) 2026-02-25T08:46:05.096 INFO:tasks.rados.rados.0.trial039.stdout:1007: done (3 left) 2026-02-25T08:46:05.096 INFO:tasks.rados.rados.0.trial039.stdout:1008: done (2 left) 2026-02-25T08:46:05.096 INFO:tasks.rados.rados.0.trial039.stdout:1009: done (1 left) 2026-02-25T08:46:05.096 INFO:tasks.rados.rados.0.trial039.stdout:1012: done (0 left) 2026-02-25T08:46:05.096 INFO:tasks.rados.rados.0.trial039.stdout:1013: snap_create 2026-02-25T08:46:06.097 INFO:tasks.rados.rados.0.trial039.stdout:1013: done (0 left) 2026-02-25T08:46:06.097 INFO:tasks.rados.rados.0.trial039.stdout:1014: snap_create 2026-02-25T08:46:06.261 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:06.337 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:06.335+0000 7fec14ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:06.340 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:06.338+0000 7fec14ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:06.367 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:06.383 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:07.098 INFO:tasks.rados.rados.0.trial039.stdout:1014: done (0 left) 2026-02-25T08:46:07.098 INFO:tasks.rados.rados.0.trial039.stdout:1015: read oid 50 snap -1 2026-02-25T08:46:07.098 INFO:tasks.rados.rados.0.trial039.stdout:1015: expect (ObjNum 470 snap 122 seq_num 470) 2026-02-25T08:46:07.098 INFO:tasks.rados.rados.0.trial039.stdout:1016: read oid 1 snap -1 2026-02-25T08:46:07.098 INFO:tasks.rados.rados.0.trial039.stdout:1016: expect deleted 2026-02-25T08:46:07.099 INFO:tasks.rados.rados.0.trial039.stdout:1017: write (excl) oid 38 current snap is 131 2026-02-25T08:46:07.099 INFO:tasks.rados.rados.0.trial039.stdout:1017: seq_num 483 ranges {517781=475506,1731038=386743} 2026-02-25T08:46:07.101 INFO:tasks.rados.rados.0.trial039.stdout:1017: writing trial03942481-38 from 517781 to 993287 tid 1 2026-02-25T08:46:07.106 INFO:tasks.rados.rados.0.trial039.stdout:1017: writing trial03942481-38 from 1731038 to 2117781 tid 2 2026-02-25T08:46:07.106 INFO:tasks.rados.rados.0.trial039.stdout:1018: write oid 23 current snap is 131 2026-02-25T08:46:07.106 INFO:tasks.rados.rados.0.trial039.stdout:1018: seq_num 484 ranges {534588=550755,1789520=345068} 2026-02-25T08:46:07.110 INFO:tasks.rados.rados.0.trial039.stdout:1018: writing trial03942481-23 from 534588 to 1085343 tid 1 2026-02-25T08:46:07.115 INFO:tasks.rados.rados.0.trial039.stdout:1018: writing trial03942481-23 from 1789520 to 2134588 tid 2 2026-02-25T08:46:07.115 INFO:tasks.rados.rados.0.trial039.stdout:1019: read oid 20 snap -1 2026-02-25T08:46:07.115 INFO:tasks.rados.rados.0.trial039.stdout:1019: expect (ObjNum 468 snap 120 seq_num 468) 2026-02-25T08:46:07.115 INFO:tasks.rados.rados.0.trial039.stdout:1020: snap_create 2026-02-25T08:46:07.130 INFO:tasks.rados.rados.0.trial039.stdout:1017: finishing write tid 1 to trial03942481-38 2026-02-25T08:46:07.139 INFO:tasks.rados.rados.0.trial039.stdout:1015: expect (ObjNum 470 snap 122 seq_num 470) 2026-02-25T08:46:07.253 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 3190971 2026-02-25T08:46:07.263 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:07.333 INFO:tasks.rados.rados.0.trial039.stdout:1019: expect (ObjNum 468 snap 120 seq_num 468) 2026-02-25T08:46:07.333 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:07.431 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1851390 2026-02-25T08:46:07.491 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1644090 2026-02-25T08:46:07.557 INFO:tasks.rados.rados.0.trial039.stdout:1017: finishing write tid 2 to trial03942481-38 2026-02-25T08:46:07.557 INFO:tasks.rados.rados.0.trial039.stdout:1017: finishing write tid 3 to trial03942481-38 2026-02-25T08:46:07.557 INFO:tasks.rados.rados.0.trial039.stdout:1017: finishing write tid 4 to trial03942481-38 2026-02-25T08:46:07.557 INFO:tasks.rados.rados.0.trial039.stdout:1017: oid 38 updating version 0 to 190 2026-02-25T08:46:07.557 INFO:tasks.rados.rados.0.trial039.stdout:1017: oid 38 version 190 is already newer than 190 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:1017: oid 38 version 190 is already newer than 188 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 38 v 190 (ObjNum 483 snap 131 seq_num 483) dirty exists 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:1017: left oid 38 (ObjNum 483 snap 131 seq_num 483) 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:1018: finishing write tid 1 to trial03942481-23 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:1018: finishing write tid 2 to trial03942481-23 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:1018: finishing write tid 3 to trial03942481-23 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:1018: finishing write tid 4 to trial03942481-23 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:1018: oid 23 updating version 0 to 201 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:1018: oid 23 version 201 is already newer than 201 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:1018: oid 23 version 201 is already newer than 201 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 23 v 201 (ObjNum 484 snap 131 seq_num 484) dirty exists 2026-02-25T08:46:07.558 INFO:tasks.rados.rados.0.trial039.stdout:1018: left oid 23 (ObjNum 484 snap 131 seq_num 484) 2026-02-25T08:46:07.891 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:07.962 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:07.966 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:08.098 INFO:tasks.rados.rados.0.trial039.stdout:1015: done (5 left) 2026-02-25T08:46:08.098 INFO:tasks.rados.rados.0.trial039.stdout:1016: done (4 left) 2026-02-25T08:46:08.098 INFO:tasks.rados.rados.0.trial039.stdout:1017: done (3 left) 2026-02-25T08:46:08.098 INFO:tasks.rados.rados.0.trial039.stdout:1018: done (2 left) 2026-02-25T08:46:08.098 INFO:tasks.rados.rados.0.trial039.stdout:1019: done (1 left) 2026-02-25T08:46:08.099 INFO:tasks.rados.rados.0.trial039.stdout:1020: done (0 left) 2026-02-25T08:46:08.099 INFO:tasks.rados.rados.0.trial039.stdout:1021: snap_create 2026-02-25T08:46:08.384 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:08.463 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:08.461+0000 7f8607b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:08.466 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:08.465+0000 7f8607b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:08.493 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:08.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:09.099 INFO:tasks.rados.rados.0.trial039.stdout:1021: done (0 left) 2026-02-25T08:46:09.100 INFO:tasks.rados.rados.0.trial039.stdout:1022: delete oid 25 current snap is 133 2026-02-25T08:46:09.148 INFO:tasks.rados.rados.0.trial039.stdout:1022: done (0 left) 2026-02-25T08:46:09.148 INFO:tasks.rados.rados.0.trial039.stdout:1023: write oid 35 current snap is 133 2026-02-25T08:46:09.148 INFO:tasks.rados.rados.0.trial039.stdout:1023: seq_num 486 ranges {568202=701253,1906484=261718} 2026-02-25T08:46:09.153 INFO:tasks.rados.rados.0.trial039.stdout:1023: writing trial03942481-35 from 568202 to 1269455 tid 1 2026-02-25T08:46:09.157 INFO:tasks.rados.rados.0.trial039.stdout:1023: writing trial03942481-35 from 1906484 to 2168202 tid 2 2026-02-25T08:46:09.157 INFO:tasks.rados.rados.0.trial039.stdout:1024: read oid 12 snap -1 2026-02-25T08:46:09.157 INFO:tasks.rados.rados.0.trial039.stdout:1024: expect deleted 2026-02-25T08:46:09.157 INFO:tasks.rados.rados.0.trial039.stdout:1025: write oid 39 current snap is 133 2026-02-25T08:46:09.157 INFO:tasks.rados.rados.0.trial039.stdout:1025: seq_num 487 ranges {585009=492855,1564966=620043} 2026-02-25T08:46:09.161 INFO:tasks.rados.rados.0.trial039.stdout:1025: writing trial03942481-39 from 585009 to 1077864 tid 1 2026-02-25T08:46:09.166 INFO:tasks.rados.rados.0.trial039.stdout:1025: writing trial03942481-39 from 1564966 to 2185009 tid 2 2026-02-25T08:46:09.166 INFO:tasks.rados.rados.0.trial039.stdout:1026: read oid 41 snap -1 2026-02-25T08:46:09.166 INFO:tasks.rados.rados.0.trial039.stdout:1026: expect deleted 2026-02-25T08:46:09.166 INFO:tasks.rados.rados.0.trial039.stdout:1027: read oid 9 snap 130 2026-02-25T08:46:09.166 INFO:tasks.rados.rados.0.trial039.stdout:1027: expect (ObjNum 464 snap 120 seq_num 464) 2026-02-25T08:46:09.167 INFO:tasks.rados.rados.0.trial039.stdout:1028: read oid 50 snap 132 2026-02-25T08:46:09.167 INFO:tasks.rados.rados.0.trial039.stdout:1028: expect (ObjNum 470 snap 122 seq_num 470) 2026-02-25T08:46:09.167 INFO:tasks.rados.rados.0.trial039.stdout:1029: delete oid 33 current snap is 133 2026-02-25T08:46:09.197 INFO:tasks.rados.rados.0.trial039.stdout:1028: expect (ObjNum 470 snap 122 seq_num 470) 2026-02-25T08:46:09.197 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 163066 2026-02-25T08:46:09.285 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1933094 2026-02-25T08:46:09.327 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:09.396 INFO:tasks.rados.rados.0.trial039.stdout:1027: expect (ObjNum 464 snap 120 seq_num 464) 2026-02-25T08:46:09.396 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:09.553 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1783262 2026-02-25T08:46:09.588 INFO:tasks.rados.rados.0.trial039.stdout:1025: finishing write tid 1 to trial03942481-39 2026-02-25T08:46:09.588 INFO:tasks.rados.rados.0.trial039.stdout:1025: finishing write tid 2 to trial03942481-39 2026-02-25T08:46:09.588 INFO:tasks.rados.rados.0.trial039.stdout:1024: done (6 left) 2026-02-25T08:46:09.588 INFO:tasks.rados.rados.0.trial039.stdout:1026: done (5 left) 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1027: done (4 left) 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1028: done (3 left) 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1029: done (2 left) 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1030: read oid 25 snap 132 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1030: expect (ObjNum 460 snap 120 seq_num 460) 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1031: write oid 22 current snap is 133 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1025: finishing write tid 3 to trial03942481-39 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1025: finishing write tid 4 to trial03942481-39 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1025: oid 39 updating version 0 to 371 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1025: oid 39 version 371 is already newer than 371 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:1025: oid 39 version 371 is already newer than 371 2026-02-25T08:46:09.589 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 39 v 371 (ObjNum 487 snap 133 seq_num 487) dirty exists 2026-02-25T08:46:09.590 INFO:tasks.rados.rados.0.trial039.stdout:1025: left oid 39 (ObjNum 487 snap 133 seq_num 487) 2026-02-25T08:46:09.590 INFO:tasks.rados.rados.0.trial039.stdout:1023: finishing write tid 1 to trial03942481-35 2026-02-25T08:46:09.590 INFO:tasks.rados.rados.0.trial039.stdout:1023: finishing write tid 2 to trial03942481-35 2026-02-25T08:46:09.590 INFO:tasks.rados.rados.0.trial039.stdout:1031: seq_num 489 ranges {618623=643353,1681930=536693} 2026-02-25T08:46:09.593 INFO:tasks.rados.rados.0.trial039.stdout:1031: writing trial03942481-22 from 618623 to 1261976 tid 1 2026-02-25T08:46:09.597 INFO:tasks.rados.rados.0.trial039.stdout:1031: writing trial03942481-22 from 1681930 to 2218623 tid 2 2026-02-25T08:46:09.597 INFO:tasks.rados.rados.0.trial039.stdout:1025: done (3 left) 2026-02-25T08:46:09.597 INFO:tasks.rados.rados.0.trial039.stdout:1032: write (excl) oid 5 current snap is 133 2026-02-25T08:46:09.597 INFO:tasks.rados.rados.0.trial039.stdout:1032: seq_num 490 ranges {635430=718602,2140412=95018} 2026-02-25T08:46:09.602 INFO:tasks.rados.rados.0.trial039.stdout:1032: writing trial03942481-5 from 635430 to 1354032 tid 1 2026-02-25T08:46:09.605 INFO:tasks.rados.rados.0.trial039.stdout:1032: writing trial03942481-5 from 2140412 to 2235430 tid 2 2026-02-25T08:46:09.606 INFO:tasks.rados.rados.0.trial039.stdout:1033: snap_create 2026-02-25T08:46:09.606 INFO:tasks.rados.rados.0.trial039.stdout:1023: finishing write tid 3 to trial03942481-35 2026-02-25T08:46:09.606 INFO:tasks.rados.rados.0.trial039.stdout:1023: finishing write tid 4 to trial03942481-35 2026-02-25T08:46:09.606 INFO:tasks.rados.rados.0.trial039.stdout:1023: oid 35 updating version 0 to 154 2026-02-25T08:46:09.606 INFO:tasks.rados.rados.0.trial039.stdout:1023: oid 35 version 154 is already newer than 154 2026-02-25T08:46:09.606 INFO:tasks.rados.rados.0.trial039.stdout:1023: oid 35 version 154 is already newer than 154 2026-02-25T08:46:09.606 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 35 v 154 (ObjNum 486 snap 133 seq_num 486) dirty exists 2026-02-25T08:46:09.606 INFO:tasks.rados.rados.0.trial039.stdout:1023: left oid 35 (ObjNum 486 snap 133 seq_num 486) 2026-02-25T08:46:09.617 INFO:tasks.rados.rados.0.trial039.stdout:1030: expect (ObjNum 460 snap 120 seq_num 460) 2026-02-25T08:46:09.672 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:09.730 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:09.789 INFO:tasks.rados.rados.0.trial039.stdout:1031: finishing write tid 1 to trial03942481-22 2026-02-25T08:46:09.789 INFO:tasks.rados.rados.0.trial039.stdout:1032: finishing write tid 1 to trial03942481-5 2026-02-25T08:46:09.789 INFO:tasks.rados.rados.0.trial039.stdout:1031: finishing write tid 2 to trial03942481-22 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1032: finishing write tid 2 to trial03942481-5 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1032: finishing write tid 3 to trial03942481-5 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1032: finishing write tid 4 to trial03942481-5 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1032: oid 5 updating version 0 to 205 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1032: oid 5 version 205 is already newer than 205 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1032: oid 5 version 205 is already newer than 203 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 5 v 205 (ObjNum 490 snap 133 seq_num 490) dirty exists 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1032: left oid 5 (ObjNum 490 snap 133 seq_num 490) 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1031: finishing write tid 3 to trial03942481-22 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1031: finishing write tid 4 to trial03942481-22 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1031: oid 22 updating version 0 to 56 2026-02-25T08:46:09.790 INFO:tasks.rados.rados.0.trial039.stdout:1031: oid 22 version 56 is already newer than 56 2026-02-25T08:46:09.791 INFO:tasks.rados.rados.0.trial039.stdout:1031: oid 22 version 56 is already newer than 56 2026-02-25T08:46:09.791 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 22 v 56 (ObjNum 489 snap 133 seq_num 489) dirty exists 2026-02-25T08:46:09.791 INFO:tasks.rados.rados.0.trial039.stdout:1031: left oid 22 (ObjNum 489 snap 133 seq_num 489) 2026-02-25T08:46:10.160 INFO:tasks.rados.rados.0.trial039.stdout:1023: done (4 left) 2026-02-25T08:46:10.160 INFO:tasks.rados.rados.0.trial039.stdout:1030: done (3 left) 2026-02-25T08:46:10.160 INFO:tasks.rados.rados.0.trial039.stdout:1031: done (2 left) 2026-02-25T08:46:10.160 INFO:tasks.rados.rados.0.trial039.stdout:1032: done (1 left) 2026-02-25T08:46:10.160 INFO:tasks.rados.rados.0.trial039.stdout:1033: done (0 left) 2026-02-25T08:46:10.160 INFO:tasks.rados.rados.0.trial039.stdout:1034: write oid 43 current snap is 134 2026-02-25T08:46:10.160 INFO:tasks.rados.rados.0.trial039.stdout:1034: seq_num 491 ranges {652237=793851,2198894=53343} 2026-02-25T08:46:10.165 INFO:tasks.rados.rados.0.trial039.stdout:1034: writing trial03942481-43 from 652237 to 1446088 tid 1 2026-02-25T08:46:10.168 INFO:tasks.rados.rados.0.trial039.stdout:1034: writing trial03942481-43 from 2198894 to 2252237 tid 2 2026-02-25T08:46:10.168 INFO:tasks.rados.rados.0.trial039.stdout:1035: snap_create 2026-02-25T08:46:10.207 INFO:tasks.rados.rados.0.trial039.stdout:1034: finishing write tid 1 to trial03942481-43 2026-02-25T08:46:10.215 INFO:tasks.rados.rados.0.trial039.stdout:1034: finishing write tid 2 to trial03942481-43 2026-02-25T08:46:10.222 INFO:tasks.rados.rados.0.trial039.stdout:1034: finishing write tid 3 to trial03942481-43 2026-02-25T08:46:10.224 INFO:tasks.rados.rados.0.trial039.stdout:1034: finishing write tid 4 to trial03942481-43 2026-02-25T08:46:10.224 INFO:tasks.rados.rados.0.trial039.stdout:1034: oid 43 updating version 0 to 162 2026-02-25T08:46:10.224 INFO:tasks.rados.rados.0.trial039.stdout:1034: oid 43 version 162 is already newer than 162 2026-02-25T08:46:10.225 INFO:tasks.rados.rados.0.trial039.stdout:1034: oid 43 version 162 is already newer than 162 2026-02-25T08:46:10.225 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 43 v 162 (ObjNum 491 snap 134 seq_num 491) dirty exists 2026-02-25T08:46:10.225 INFO:tasks.rados.rados.0.trial039.stdout:1034: left oid 43 (ObjNum 491 snap 134 seq_num 491) 2026-02-25T08:46:10.512 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:10.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~275s 2026-02-25T08:46:10.588 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:10.585+0000 7fd5e550b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:10.590 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:10.589+0000 7fd5e550b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:10.619 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:10.635 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:10.636 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:11.166 INFO:tasks.rados.rados.0.trial039.stdout:1034: done (1 left) 2026-02-25T08:46:11.167 INFO:tasks.rados.rados.0.trial039.stdout:1035: done (0 left) 2026-02-25T08:46:11.167 INFO:tasks.rados.rados.0.trial039.stdout:1036: read oid 22 snap -1 2026-02-25T08:46:11.167 INFO:tasks.rados.rados.0.trial039.stdout:1036: expect (ObjNum 489 snap 133 seq_num 489) 2026-02-25T08:46:11.167 INFO:tasks.rados.rados.0.trial039.stdout:1037: read oid 14 snap 131 2026-02-25T08:46:11.167 INFO:tasks.rados.rados.0.trial039.stdout:1037: expect deleted 2026-02-25T08:46:11.167 INFO:tasks.rados.rados.0.trial039.stdout:1038: write oid 45 current snap is 135 2026-02-25T08:46:11.167 INFO:tasks.rados.rados.0.trial039.stdout:1038: seq_num 492 ranges {669044=469100,1741023=459161} 2026-02-25T08:46:11.170 INFO:tasks.rados.rados.0.trial039.stdout:1038: writing trial03942481-45 from 669044 to 1138144 tid 1 2026-02-25T08:46:11.175 INFO:tasks.rados.rados.0.trial039.stdout:1038: writing trial03942481-45 from 1741023 to 2200184 tid 2 2026-02-25T08:46:11.175 INFO:tasks.rados.rados.0.trial039.stdout:1039: delete oid 10 current snap is 135 2026-02-25T08:46:11.189 INFO:tasks.rados.rados.0.trial039.stdout:1037: done (3 left) 2026-02-25T08:46:11.189 INFO:tasks.rados.rados.0.trial039.stdout:1039: done (2 left) 2026-02-25T08:46:11.189 INFO:tasks.rados.rados.0.trial039.stdout:1040: snap_remove snap 128 2026-02-25T08:46:12.168 INFO:tasks.rados.rados.0.trial039.stdout:1040: done (2 left) 2026-02-25T08:46:12.169 INFO:tasks.rados.rados.0.trial039.stdout:1041: write (excl) oid 14 current snap is 135 2026-02-25T08:46:12.169 INFO:tasks.rados.rados.0.trial039.stdout:1041: seq_num 494 ranges {702658=619598,1857987=444671} 2026-02-25T08:46:12.173 INFO:tasks.rados.rados.0.trial039.stdout:1041: writing trial03942481-14 from 702658 to 1322256 tid 1 2026-02-25T08:46:12.177 INFO:tasks.rados.rados.0.trial039.stdout:1041: writing trial03942481-14 from 1857987 to 2302658 tid 2 2026-02-25T08:46:12.177 INFO:tasks.rados.rados.0.trial039.stdout:1042: snap_remove snap 132 2026-02-25T08:46:12.177 INFO:tasks.rados.rados.0.trial039.stdout:1036: expect (ObjNum 489 snap 133 seq_num 489) 2026-02-25T08:46:12.177 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 554735 2026-02-25T08:46:12.204 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:12.239 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 941465 2026-02-25T08:46:12.258 INFO:tasks.rados.rados.0.trial039.stdout:1038: finishing write tid 1 to trial03942481-45 2026-02-25T08:46:12.637 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:12.713 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:12.711+0000 7f73721dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:12.715 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:12.714+0000 7f73721dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:12.748 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:12.766 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:12.967 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:13.040 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:13.044 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:13.096 INFO:tasks.rados.rados.0.trial039.stdout:1036: done (3 left) 2026-02-25T08:46:13.096 INFO:tasks.rados.rados.0.trial039.stdout:1042: done (2 left) 2026-02-25T08:46:13.097 INFO:tasks.rados.rados.0.trial039.stdout:1043: write (excl) oid 43 current snap is 135 2026-02-25T08:46:13.097 INFO:tasks.rados.rados.0.trial039.stdout:1038: finishing write tid 2 to trial03942481-45 2026-02-25T08:46:13.097 INFO:tasks.rados.rados.0.trial039.stdout:1043: seq_num 495 ranges {719465=411200,1632822=686643} 2026-02-25T08:46:13.100 INFO:tasks.rados.rados.0.trial039.stdout:1043: writing trial03942481-43 from 719465 to 1130665 tid 1 2026-02-25T08:46:13.106 INFO:tasks.rados.rados.0.trial039.stdout:1043: writing trial03942481-43 from 1632822 to 2319465 tid 2 2026-02-25T08:46:13.106 INFO:tasks.rados.rados.0.trial039.stdout:1044: snap_create 2026-02-25T08:46:13.106 INFO:tasks.rados.rados.0.trial039.stdout:1038: finishing write tid 3 to trial03942481-45 2026-02-25T08:46:13.106 INFO:tasks.rados.rados.0.trial039.stdout:1038: finishing write tid 4 to trial03942481-45 2026-02-25T08:46:13.106 INFO:tasks.rados.rados.0.trial039.stdout:1038: oid 45 updating version 0 to 211 2026-02-25T08:46:13.106 INFO:tasks.rados.rados.0.trial039.stdout:1038: oid 45 version 211 is already newer than 211 2026-02-25T08:46:13.106 INFO:tasks.rados.rados.0.trial039.stdout:1038: oid 45 version 211 is already newer than 211 2026-02-25T08:46:13.106 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 45 v 211 (ObjNum 492 snap 135 seq_num 492) dirty exists 2026-02-25T08:46:13.106 INFO:tasks.rados.rados.0.trial039.stdout:1038: left oid 45 (ObjNum 492 snap 135 seq_num 492) 2026-02-25T08:46:13.107 INFO:tasks.rados.rados.0.trial039.stdout:1041: finishing write tid 1 to trial03942481-14 2026-02-25T08:46:13.107 INFO:tasks.rados.rados.0.trial039.stdout:1041: finishing write tid 2 to trial03942481-14 2026-02-25T08:46:13.107 INFO:tasks.rados.rados.0.trial039.stdout:1041: finishing write tid 3 to trial03942481-14 2026-02-25T08:46:13.107 INFO:tasks.rados.rados.0.trial039.stdout:1041: finishing write tid 4 to trial03942481-14 2026-02-25T08:46:13.107 INFO:tasks.rados.rados.0.trial039.stdout:1041: oid 14 updating version 0 to 212 2026-02-25T08:46:13.107 INFO:tasks.rados.rados.0.trial039.stdout:1041: oid 14 updating version 212 to 214 2026-02-25T08:46:13.107 INFO:tasks.rados.rados.0.trial039.stdout:1041: oid 14 version 214 is already newer than 214 2026-02-25T08:46:13.107 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 14 v 214 (ObjNum 494 snap 135 seq_num 494) dirty exists 2026-02-25T08:46:13.107 INFO:tasks.rados.rados.0.trial039.stdout:1041: left oid 14 (ObjNum 494 snap 135 seq_num 494) 2026-02-25T08:46:13.137 INFO:tasks.rados.rados.0.trial039.stdout:1043: finishing write tid 1 to trial03942481-43 2026-02-25T08:46:13.154 INFO:tasks.rados.rados.0.trial039.stdout:1043: finishing write tid 2 to trial03942481-43 2026-02-25T08:46:13.157 INFO:tasks.rados.rados.0.trial039.stdout:1043: finishing write tid 3 to trial03942481-43 2026-02-25T08:46:13.159 INFO:tasks.rados.rados.0.trial039.stdout:1043: finishing write tid 4 to trial03942481-43 2026-02-25T08:46:13.160 INFO:tasks.rados.rados.0.trial039.stdout:1043: oid 43 updating version 0 to 166 2026-02-25T08:46:13.160 INFO:tasks.rados.rados.0.trial039.stdout:1043: oid 43 version 166 is already newer than 164 2026-02-25T08:46:13.160 INFO:tasks.rados.rados.0.trial039.stdout:1043: oid 43 version 166 is already newer than 166 2026-02-25T08:46:13.160 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 43 v 166 (ObjNum 495 snap 135 seq_num 495) dirty exists 2026-02-25T08:46:13.160 INFO:tasks.rados.rados.0.trial039.stdout:1043: left oid 43 (ObjNum 495 snap 135 seq_num 495) 2026-02-25T08:46:14.097 INFO:tasks.rados.rados.0.trial039.stdout:1038: done (3 left) 2026-02-25T08:46:14.097 INFO:tasks.rados.rados.0.trial039.stdout:1041: done (2 left) 2026-02-25T08:46:14.098 INFO:tasks.rados.rados.0.trial039.stdout:1043: done (1 left) 2026-02-25T08:46:14.098 INFO:tasks.rados.rados.0.trial039.stdout:1044: done (0 left) 2026-02-25T08:46:14.098 INFO:tasks.rados.rados.0.trial039.stdout:1045: read oid 26 snap 135 2026-02-25T08:46:14.098 INFO:tasks.rados.rados.0.trial039.stdout:1045: expect deleted 2026-02-25T08:46:14.098 INFO:tasks.rados.rados.0.trial039.stdout:1046: delete oid 32 current snap is 136 2026-02-25T08:46:14.115 INFO:tasks.rados.rados.0.trial039.stdout:1046: done (1 left) 2026-02-25T08:46:14.116 INFO:tasks.rados.rados.0.trial039.stdout:1047: snap_create 2026-02-25T08:46:14.767 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:14.845 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:14.842+0000 7f5580c8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:14.847 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:14.846+0000 7f5580c8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:14.874 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:14.891 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:15.096 INFO:tasks.rados.rados.0.trial039.stdout:1045: done (1 left) 2026-02-25T08:46:15.096 INFO:tasks.rados.rados.0.trial039.stdout:1047: done (0 left) 2026-02-25T08:46:15.097 INFO:tasks.rados.rados.0.trial039.stdout:1048: snap_remove snap 125 2026-02-25T08:46:16.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~281s 2026-02-25T08:46:16.099 INFO:tasks.rados.rados.0.trial039.stdout:1048: done (0 left) 2026-02-25T08:46:16.099 INFO:tasks.rados.rados.0.trial039.stdout:1049: read oid 36 snap -1 2026-02-25T08:46:16.099 INFO:tasks.rados.rados.0.trial039.stdout:1049: expect deleted 2026-02-25T08:46:16.099 INFO:tasks.rados.rados.0.trial039.stdout:1050: snap_remove snap 130 2026-02-25T08:46:16.893 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:16.969 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:16.967+0000 7f994cac2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:16.972 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:16.971+0000 7f994cac2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:16.999 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:17.016 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:17.098 INFO:tasks.rados.rados.0.trial039.stdout:1050: done (1 left) 2026-02-25T08:46:17.098 INFO:tasks.rados.rados.0.trial039.stdout:1051: write oid 37 current snap is 137 2026-02-25T08:46:17.099 INFO:tasks.rados.rados.0.trial039.stdout:1051: seq_num 497 ranges {753079=561698,2033433=319646} 2026-02-25T08:46:17.103 INFO:tasks.rados.rados.0.trial039.stdout:1051: writing trial03942481-37 from 753079 to 1314777 tid 1 2026-02-25T08:46:17.107 INFO:tasks.rados.rados.0.trial039.stdout:1051: writing trial03942481-37 from 2033433 to 2353079 tid 2 2026-02-25T08:46:17.108 INFO:tasks.rados.rados.0.trial039.stdout:1052: read oid 17 snap -1 2026-02-25T08:46:17.108 INFO:tasks.rados.rados.0.trial039.stdout:1052: expect (ObjNum 475 snap 124 seq_num 475) 2026-02-25T08:46:17.108 INFO:tasks.rados.rados.0.trial039.stdout:1053: snap_create 2026-02-25T08:46:17.133 INFO:tasks.rados.rados.0.trial039.stdout:1051: finishing write tid 1 to trial03942481-37 2026-02-25T08:46:17.146 INFO:tasks.rados.rados.0.trial039.stdout:1051: finishing write tid 2 to trial03942481-37 2026-02-25T08:46:17.153 INFO:tasks.rados.rados.0.trial039.stdout:1051: finishing write tid 3 to trial03942481-37 2026-02-25T08:46:17.155 INFO:tasks.rados.rados.0.trial039.stdout:1051: finishing write tid 4 to trial03942481-37 2026-02-25T08:46:17.155 INFO:tasks.rados.rados.0.trial039.stdout:1051: oid 37 updating version 0 to 204 2026-02-25T08:46:17.155 INFO:tasks.rados.rados.0.trial039.stdout:1051: oid 37 version 204 is already newer than 204 2026-02-25T08:46:17.156 INFO:tasks.rados.rados.0.trial039.stdout:1051: oid 37 version 204 is already newer than 204 2026-02-25T08:46:17.156 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 37 v 204 (ObjNum 497 snap 137 seq_num 497) dirty exists 2026-02-25T08:46:17.156 INFO:tasks.rados.rados.0.trial039.stdout:1051: left oid 37 (ObjNum 497 snap 137 seq_num 497) 2026-02-25T08:46:17.166 INFO:tasks.rados.rados.0.trial039.stdout:1052: expect (ObjNum 475 snap 124 seq_num 475) 2026-02-25T08:46:17.177 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1581652 2026-02-25T08:46:17.317 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 3757186 2026-02-25T08:46:18.044 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:18.100 INFO:tasks.rados.rados.0.trial039.stdout:1049: done (3 left) 2026-02-25T08:46:18.100 INFO:tasks.rados.rados.0.trial039.stdout:1051: done (2 left) 2026-02-25T08:46:18.100 INFO:tasks.rados.rados.0.trial039.stdout:1052: done (1 left) 2026-02-25T08:46:18.100 INFO:tasks.rados.rados.0.trial039.stdout:1053: done (0 left) 2026-02-25T08:46:18.101 INFO:tasks.rados.rados.0.trial039.stdout:1054: read oid 43 snap 134 2026-02-25T08:46:18.101 INFO:tasks.rados.rados.0.trial039.stdout:1054: expect (ObjNum 491 snap 134 seq_num 491) 2026-02-25T08:46:18.101 INFO:tasks.rados.rados.0.trial039.stdout:1055: read oid 5 snap -1 2026-02-25T08:46:18.101 INFO:tasks.rados.rados.0.trial039.stdout:1055: expect (ObjNum 490 snap 133 seq_num 490) 2026-02-25T08:46:18.101 INFO:tasks.rados.rados.0.trial039.stdout:1056: read oid 23 snap -1 2026-02-25T08:46:18.101 INFO:tasks.rados.rados.0.trial039.stdout:1056: expect (ObjNum 484 snap 131 seq_num 484) 2026-02-25T08:46:18.101 INFO:tasks.rados.rados.0.trial039.stdout:1057: snap_create 2026-02-25T08:46:18.119 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:18.123 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:18.138 INFO:tasks.rados.rados.0.trial039.stdout:1055: expect (ObjNum 490 snap 133 seq_num 490) 2026-02-25T08:46:18.142 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1459118 2026-02-25T08:46:18.163 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:18.195 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:18.227 INFO:tasks.rados.rados.0.trial039.stdout:1054: expect (ObjNum 491 snap 134 seq_num 491) 2026-02-25T08:46:18.227 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:18.340 INFO:tasks.rados.rados.0.trial039.stdout:1056: expect (ObjNum 484 snap 131 seq_num 484) 2026-02-25T08:46:18.365 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 1533205 2026-02-25T08:46:18.377 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 6581 2026-02-25T08:46:19.017 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:19.095 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:19.092+0000 7f1fb26cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:19.097 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:19.096+0000 7f1fb26cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:19.100 INFO:tasks.rados.rados.0.trial039.stdout:1054: done (3 left) 2026-02-25T08:46:19.100 INFO:tasks.rados.rados.0.trial039.stdout:1055: done (2 left) 2026-02-25T08:46:19.100 INFO:tasks.rados.rados.0.trial039.stdout:1056: done (1 left) 2026-02-25T08:46:19.100 INFO:tasks.rados.rados.0.trial039.stdout:1057: done (0 left) 2026-02-25T08:46:19.100 INFO:tasks.rados.rados.0.trial039.stdout:1058: read oid 33 snap 136 2026-02-25T08:46:19.100 INFO:tasks.rados.rados.0.trial039.stdout:1058: expect deleted 2026-02-25T08:46:19.100 INFO:tasks.rados.rados.0.trial039.stdout:1059: write (excl) oid 12 current snap is 139 2026-02-25T08:46:19.100 INFO:tasks.rados.rados.0.trial039.stdout:1059: seq_num 498 ranges {769886=636947,2091915=277971} 2026-02-25T08:46:19.105 INFO:tasks.rados.rados.0.trial039.stdout:1059: writing trial03942481-12 from 769886 to 1406833 tid 1 2026-02-25T08:46:19.109 INFO:tasks.rados.rados.0.trial039.stdout:1059: writing trial03942481-12 from 2091915 to 2369886 tid 2 2026-02-25T08:46:19.109 INFO:tasks.rados.rados.0.trial039.stdout:1060: write (excl) oid 46 current snap is 139 2026-02-25T08:46:19.109 INFO:tasks.rados.rados.0.trial039.stdout:1060: seq_num 499 ranges {786693=712196,2150397=236296} 2026-02-25T08:46:19.115 INFO:tasks.rados.rados.0.trial039.stdout:1060: writing trial03942481-46 from 786693 to 1498889 tid 1 2026-02-25T08:46:19.118 INFO:tasks.rados.rados.0.trial039.stdout:1060: writing trial03942481-46 from 2150397 to 2386693 tid 2 2026-02-25T08:46:19.119 INFO:tasks.rados.rados.0.trial039.stdout:1061: write (excl) oid 19 current snap is 139 2026-02-25T08:46:19.119 INFO:tasks.rados.rados.0.trial039.stdout:1061: seq_num 500 ranges {403500=787445,1692526=473837} 2026-02-25T08:46:19.123 INFO:tasks.rados.rados.0.trial039.stdout:1061: writing trial03942481-19 from 403500 to 1190945 tid 1 2026-02-25T08:46:19.127 INFO:tasks.rados.rados.0.trial039.stdout:1061: writing trial03942481-19 from 1692526 to 2166363 tid 2 2026-02-25T08:46:19.127 INFO:tasks.rados.rados.0.trial039.stdout:1062: write oid 5 current snap is 139 2026-02-25T08:46:19.127 INFO:tasks.rados.rados.0.trial039.stdout:1062: seq_num 501 ranges {420307=462694,1351008=617495} 2026-02-25T08:46:19.130 INFO:tasks.rados.rados.0.trial039.stdout:1062: writing trial03942481-5 from 420307 to 883001 tid 1 2026-02-25T08:46:19.135 INFO:tasks.rados.rados.0.trial039.stdout:1062: writing trial03942481-5 from 1351008 to 1968503 tid 2 2026-02-25T08:46:19.135 INFO:tasks.rados.rados.0.trial039.stdout:1063: write oid 28 current snap is 139 2026-02-25T08:46:19.135 INFO:tasks.rados.rados.0.trial039.stdout:1063: seq_num 502 ranges {437114=654296,1525843=477506} 2026-02-25T08:46:19.139 INFO:tasks.rados.rados.0.trial039.stdout:1063: writing trial03942481-28 from 437114 to 1091410 tid 1 2026-02-25T08:46:19.142 INFO:tasks.rados.rados.0.trial039.stdout:1063: writing trial03942481-28 from 1525843 to 2003349 tid 2 2026-02-25T08:46:19.143 INFO:tasks.rados.rados.0.trial039.stdout:1058: done (5 left) 2026-02-25T08:46:19.143 INFO:tasks.rados.rados.0.trial039.stdout:1064: delete oid 38 current snap is 139 2026-02-25T08:46:19.144 INFO:tasks.rados.rados.0.trial039.stdout:1059: finishing write tid 1 to trial03942481-12 2026-02-25T08:46:19.147 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:19.174 INFO:tasks.rados.rados.0.trial039.stdout:1059: finishing write tid 2 to trial03942481-12 2026-02-25T08:46:19.176 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:19.176 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:19.182 INFO:tasks.rados.rados.0.trial039.stdout:1060: finishing write tid 1 to trial03942481-46 2026-02-25T08:46:19.189 INFO:tasks.rados.rados.0.trial039.stdout:1059: finishing write tid 3 to trial03942481-12 2026-02-25T08:46:19.193 INFO:tasks.rados.rados.0.trial039.stdout:1059: finishing write tid 4 to trial03942481-12 2026-02-25T08:46:19.193 INFO:tasks.rados.rados.0.trial039.stdout:1059: oid 12 updating version 0 to 57 2026-02-25T08:46:19.193 INFO:tasks.rados.rados.0.trial039.stdout:1059: oid 12 version 57 is already newer than 55 2026-02-25T08:46:19.193 INFO:tasks.rados.rados.0.trial039.stdout:1059: oid 12 version 57 is already newer than 57 2026-02-25T08:46:19.193 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 12 v 57 (ObjNum 498 snap 139 seq_num 498) dirty exists 2026-02-25T08:46:19.194 INFO:tasks.rados.rados.0.trial039.stdout:1059: left oid 12 (ObjNum 498 snap 139 seq_num 498) 2026-02-25T08:46:19.207 INFO:tasks.rados.rados.0.trial039.stdout:1060: finishing write tid 2 to trial03942481-46 2026-02-25T08:46:19.212 INFO:tasks.rados.rados.0.trial039.stdout:1059: done (5 left) 2026-02-25T08:46:19.212 INFO:tasks.rados.rados.0.trial039.stdout:1064: done (4 left) 2026-02-25T08:46:19.212 INFO:tasks.rados.rados.0.trial039.stdout:1065: snap_create 2026-02-25T08:46:19.221 INFO:tasks.rados.rados.0.trial039.stdout:1061: finishing write tid 1 to trial03942481-19 2026-02-25T08:46:19.226 INFO:tasks.rados.rados.0.trial039.stdout:1060: finishing write tid 3 to trial03942481-46 2026-02-25T08:46:19.229 INFO:tasks.rados.rados.0.trial039.stdout:1060: finishing write tid 4 to trial03942481-46 2026-02-25T08:46:19.230 INFO:tasks.rados.rados.0.trial039.stdout:1060: oid 46 updating version 0 to 223 2026-02-25T08:46:19.230 INFO:tasks.rados.rados.0.trial039.stdout:1060: oid 46 version 223 is already newer than 221 2026-02-25T08:46:19.230 INFO:tasks.rados.rados.0.trial039.stdout:1060: oid 46 version 223 is already newer than 223 2026-02-25T08:46:19.230 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 46 v 223 (ObjNum 499 snap 139 seq_num 499) dirty exists 2026-02-25T08:46:19.230 INFO:tasks.rados.rados.0.trial039.stdout:1060: left oid 46 (ObjNum 499 snap 139 seq_num 499) 2026-02-25T08:46:19.237 INFO:tasks.rados.rados.0.trial039.stdout:1061: finishing write tid 2 to trial03942481-19 2026-02-25T08:46:19.249 INFO:tasks.rados.rados.0.trial039.stdout:1061: finishing write tid 3 to trial03942481-19 2026-02-25T08:46:19.251 INFO:tasks.rados.rados.0.trial039.stdout:1061: finishing write tid 4 to trial03942481-19 2026-02-25T08:46:19.251 INFO:tasks.rados.rados.0.trial039.stdout:1061: oid 19 updating version 0 to 383 2026-02-25T08:46:19.251 INFO:tasks.rados.rados.0.trial039.stdout:1061: oid 19 version 383 is already newer than 383 2026-02-25T08:46:19.251 INFO:tasks.rados.rados.0.trial039.stdout:1061: oid 19 version 383 is already newer than 381 2026-02-25T08:46:19.251 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 19 v 383 (ObjNum 500 snap 139 seq_num 500) dirty exists 2026-02-25T08:46:19.251 INFO:tasks.rados.rados.0.trial039.stdout:1061: left oid 19 (ObjNum 500 snap 139 seq_num 500) 2026-02-25T08:46:19.257 INFO:tasks.rados.rados.0.trial039.stdout:1062: finishing write tid 1 to trial03942481-5 2026-02-25T08:46:19.270 INFO:tasks.rados.rados.0.trial039.stdout:1062: finishing write tid 2 to trial03942481-5 2026-02-25T08:46:19.276 INFO:tasks.rados.rados.0.trial039.stdout:1062: finishing write tid 3 to trial03942481-5 2026-02-25T08:46:19.278 INFO:tasks.rados.rados.0.trial039.stdout:1062: finishing write tid 4 to trial03942481-5 2026-02-25T08:46:19.278 INFO:tasks.rados.rados.0.trial039.stdout:1062: oid 5 updating version 0 to 227 2026-02-25T08:46:19.278 INFO:tasks.rados.rados.0.trial039.stdout:1062: oid 5 version 227 is already newer than 227 2026-02-25T08:46:19.278 INFO:tasks.rados.rados.0.trial039.stdout:1062: oid 5 version 227 is already newer than 227 2026-02-25T08:46:19.278 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 5 v 227 (ObjNum 501 snap 139 seq_num 501) dirty exists 2026-02-25T08:46:19.278 INFO:tasks.rados.rados.0.trial039.stdout:1062: left oid 5 (ObjNum 501 snap 139 seq_num 501) 2026-02-25T08:46:19.300 INFO:tasks.rados.rados.0.trial039.stdout:1063: finishing write tid 1 to trial03942481-28 2026-02-25T08:46:19.309 INFO:tasks.rados.rados.0.trial039.stdout:1063: finishing write tid 2 to trial03942481-28 2026-02-25T08:46:19.317 INFO:tasks.rados.rados.0.trial039.stdout:1063: finishing write tid 3 to trial03942481-28 2026-02-25T08:46:19.319 INFO:tasks.rados.rados.0.trial039.stdout:1063: finishing write tid 4 to trial03942481-28 2026-02-25T08:46:19.319 INFO:tasks.rados.rados.0.trial039.stdout:1063: oid 28 updating version 0 to 170 2026-02-25T08:46:19.319 INFO:tasks.rados.rados.0.trial039.stdout:1063: oid 28 version 170 is already newer than 170 2026-02-25T08:46:19.319 INFO:tasks.rados.rados.0.trial039.stdout:1063: oid 28 version 170 is already newer than 170 2026-02-25T08:46:19.319 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 28 v 170 (ObjNum 502 snap 139 seq_num 502) dirty exists 2026-02-25T08:46:19.319 INFO:tasks.rados.rados.0.trial039.stdout:1063: left oid 28 (ObjNum 502 snap 139 seq_num 502) 2026-02-25T08:46:20.184 INFO:tasks.rados.rados.0.trial039.stdout:1060: done (4 left) 2026-02-25T08:46:20.184 INFO:tasks.rados.rados.0.trial039.stdout:1061: done (3 left) 2026-02-25T08:46:20.184 INFO:tasks.rados.rados.0.trial039.stdout:1062: done (2 left) 2026-02-25T08:46:20.185 INFO:tasks.rados.rados.0.trial039.stdout:1063: done (1 left) 2026-02-25T08:46:20.185 INFO:tasks.rados.rados.0.trial039.stdout:1065: done (0 left) 2026-02-25T08:46:20.185 INFO:tasks.rados.rados.0.trial039.stdout:1066: write (excl) oid 45 current snap is 140 2026-02-25T08:46:20.185 INFO:tasks.rados.rados.0.trial039.stdout:1066: seq_num 504 ranges {470728=404794,1526454=481175} 2026-02-25T08:46:20.187 INFO:tasks.rados.rados.0.trial039.stdout:1066: writing trial03942481-45 from 470728 to 875522 tid 1 2026-02-25T08:46:20.191 INFO:tasks.rados.rados.0.trial039.stdout:1066: writing trial03942481-45 from 1526454 to 2007629 tid 2 2026-02-25T08:46:20.192 INFO:tasks.rados.rados.0.trial039.stdout:1067: snap_remove snap 131 2026-02-25T08:46:21.177 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:21.186 INFO:tasks.rados.rados.0.trial039.stdout:1067: done (1 left) 2026-02-25T08:46:21.186 INFO:tasks.rados.rados.0.trial039.stdout:1068: write oid 31 current snap is 140 2026-02-25T08:46:21.187 INFO:tasks.rados.rados.0.trial039.stdout:1066: finishing write tid 1 to trial03942481-45 2026-02-25T08:46:21.187 INFO:tasks.rados.rados.0.trial039.stdout:1068: seq_num 505 ranges {487535=480043,1584936=624833} 2026-02-25T08:46:21.190 INFO:tasks.rados.rados.0.trial039.stdout:1068: writing trial03942481-31 from 487535 to 967578 tid 1 2026-02-25T08:46:21.195 INFO:tasks.rados.rados.0.trial039.stdout:1068: writing trial03942481-31 from 1584936 to 2209769 tid 2 2026-02-25T08:46:21.195 INFO:tasks.rados.rados.0.trial039.stdout:1069: read oid 40 snap -1 2026-02-25T08:46:21.195 INFO:tasks.rados.rados.0.trial039.stdout:1069: expect (ObjNum 452 snap 119 seq_num 452) 2026-02-25T08:46:21.195 INFO:tasks.rados.rados.0.trial039.stdout:1070: snap_remove snap 136 2026-02-25T08:46:21.259 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:21.257+0000 7fdec5bc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:21.262 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:21.261+0000 7fdec5bc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:21.300 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:21.316 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:21.316 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:21.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2026-02-25T08:46:22.190 INFO:tasks.rados.rados.0.trial039.stdout:1070: done (3 left) 2026-02-25T08:46:22.191 INFO:tasks.rados.rados.0.trial039.stdout:1071: snap_remove snap 135 2026-02-25T08:46:23.098 INFO:tasks.rados.rados.0.trial039.stdout:1071: done (3 left) 2026-02-25T08:46:23.098 INFO:tasks.rados.rados.0.trial039.stdout:1072: delete oid 30 current snap is 140 2026-02-25T08:46:23.099 INFO:tasks.rados.rados.0.trial039.stdout:1066: finishing write tid 2 to trial03942481-45 2026-02-25T08:46:23.099 INFO:tasks.rados.rados.0.trial039.stdout:1066: finishing write tid 3 to trial03942481-45 2026-02-25T08:46:23.099 INFO:tasks.rados.rados.0.trial039.stdout:1066: finishing write tid 4 to trial03942481-45 2026-02-25T08:46:23.099 INFO:tasks.rados.rados.0.trial039.stdout:1066: oid 45 updating version 0 to 218 2026-02-25T08:46:23.099 INFO:tasks.rados.rados.0.trial039.stdout:1066: oid 45 version 218 is already newer than 218 2026-02-25T08:46:23.099 INFO:tasks.rados.rados.0.trial039.stdout:1066: oid 45 version 218 is already newer than 216 2026-02-25T08:46:23.099 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 45 v 218 (ObjNum 504 snap 140 seq_num 504) dirty exists 2026-02-25T08:46:23.099 INFO:tasks.rados.rados.0.trial039.stdout:1066: left oid 45 (ObjNum 504 snap 140 seq_num 504) 2026-02-25T08:46:23.099 INFO:tasks.rados.rados.0.trial039.stdout:1069: expect (ObjNum 452 snap 119 seq_num 452) 2026-02-25T08:46:23.124 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:23.195 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:23.199 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:23.282 INFO:tasks.rados.rados.0.trial039.stdout:1068: finishing write tid 1 to trial03942481-31 2026-02-25T08:46:23.283 INFO:tasks.rados.rados.0.trial039.stdout:1068: finishing write tid 2 to trial03942481-31 2026-02-25T08:46:23.283 INFO:tasks.rados.rados.0.trial039.stdout:1068: finishing write tid 3 to trial03942481-31 2026-02-25T08:46:23.283 INFO:tasks.rados.rados.0.trial039.stdout:1066: done (3 left) 2026-02-25T08:46:23.283 INFO:tasks.rados.rados.0.trial039.stdout:1069: done (2 left) 2026-02-25T08:46:23.283 INFO:tasks.rados.rados.0.trial039.stdout:1072: done (1 left) 2026-02-25T08:46:23.283 INFO:tasks.rados.rados.0.trial039.stdout:1073: snap_remove snap 139 2026-02-25T08:46:23.317 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:23.394 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:23.390+0000 7f29a0f5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:23.396 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:23.394+0000 7f29a0f5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:23.423 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:23.440 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:23.441 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:24.198 INFO:tasks.rados.rados.0.trial039.stdout:1073: done (1 left) 2026-02-25T08:46:24.198 INFO:tasks.rados.rados.0.trial039.stdout:1074: snap_create 2026-02-25T08:46:24.198 INFO:tasks.rados.rados.0.trial039.stdout:1068: finishing write tid 4 to trial03942481-31 2026-02-25T08:46:24.198 INFO:tasks.rados.rados.0.trial039.stdout:1068: oid 31 updating version 0 to 391 2026-02-25T08:46:24.199 INFO:tasks.rados.rados.0.trial039.stdout:1068: oid 31 version 391 is already newer than 391 2026-02-25T08:46:24.199 INFO:tasks.rados.rados.0.trial039.stdout:1068: oid 31 version 391 is already newer than 391 2026-02-25T08:46:24.199 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 391 (ObjNum 505 snap 140 seq_num 505) dirty exists 2026-02-25T08:46:24.199 INFO:tasks.rados.rados.0.trial039.stdout:1068: left oid 31 (ObjNum 505 snap 140 seq_num 505) 2026-02-25T08:46:25.201 INFO:tasks.rados.rados.0.trial039.stdout:1068: done (1 left) 2026-02-25T08:46:25.201 INFO:tasks.rados.rados.0.trial039.stdout:1074: done (0 left) 2026-02-25T08:46:25.201 INFO:tasks.rados.rados.0.trial039.stdout:1075: write (excl) oid 23 current snap is 141 2026-02-25T08:46:25.201 INFO:tasks.rados.rados.0.trial039.stdout:1075: seq_num 507 ranges {521149=630541,1701900=628502} 2026-02-25T08:46:25.205 INFO:tasks.rados.rados.0.trial039.stdout:1075: writing trial03942481-23 from 521149 to 1151690 tid 1 2026-02-25T08:46:25.211 INFO:tasks.rados.rados.0.trial039.stdout:1075: writing trial03942481-23 from 1701900 to 2330402 tid 2 2026-02-25T08:46:25.211 INFO:tasks.rados.rados.0.trial039.stdout:1076: read oid 33 snap 133 2026-02-25T08:46:25.211 INFO:tasks.rados.rados.0.trial039.stdout:1076: expect deleted 2026-02-25T08:46:25.211 INFO:tasks.rados.rados.0.trial039.stdout:1077: write oid 2 current snap is 141 2026-02-25T08:46:25.211 INFO:tasks.rados.rados.0.trial039.stdout:1077: seq_num 508 ranges {537956=705790,1644029=772160} 2026-02-25T08:46:25.215 INFO:tasks.rados.rados.0.trial039.stdout:1077: writing trial03942481-2 from 537956 to 1243746 tid 1 2026-02-25T08:46:25.220 INFO:tasks.rados.rados.0.trial039.stdout:1077: writing trial03942481-2 from 1644029 to 2416189 tid 2 2026-02-25T08:46:25.220 INFO:tasks.rados.rados.0.trial039.stdout:1078: write oid 5 current snap is 141 2026-02-25T08:46:25.220 INFO:tasks.rados.rados.0.trial039.stdout:1078: seq_num 509 ranges {554763=781039,2102511=452252} 2026-02-25T08:46:25.225 INFO:tasks.rados.rados.0.trial039.stdout:1078: writing trial03942481-5 from 554763 to 1335802 tid 1 2026-02-25T08:46:25.230 INFO:tasks.rados.rados.0.trial039.stdout:1078: writing trial03942481-5 from 2102511 to 2554763 tid 2 2026-02-25T08:46:25.230 INFO:tasks.rados.rados.0.trial039.stdout:1079: write (excl) oid 49 current snap is 141 2026-02-25T08:46:25.231 INFO:tasks.rados.rados.0.trial039.stdout:1079: seq_num 510 ranges {571570=572641,1877346=694224} 2026-02-25T08:46:25.234 INFO:tasks.rados.rados.0.trial039.stdout:1079: writing trial03942481-49 from 571570 to 1144211 tid 1 2026-02-25T08:46:25.240 INFO:tasks.rados.rados.0.trial039.stdout:1079: writing trial03942481-49 from 1877346 to 2571570 tid 2 2026-02-25T08:46:25.240 INFO:tasks.rados.rados.0.trial039.stdout:1080: snap_remove snap 134 2026-02-25T08:46:25.441 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:25.519 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:25.515+0000 7f0142acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:25.522 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:25.519+0000 7f0142acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:25.549 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:25.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:26.206 INFO:tasks.rados.rados.0.trial039.stdout:1080: done (5 left) 2026-02-25T08:46:26.206 INFO:tasks.rados.rados.0.trial039.stdout:1081: delete oid 1 current snap is 141 2026-02-25T08:46:26.206 INFO:tasks.rados.rados.0.trial039.stdout:1077: finishing write tid 1 to trial03942481-2 2026-02-25T08:46:26.206 INFO:tasks.rados.rados.0.trial039.stdout:1077: finishing write tid 2 to trial03942481-2 2026-02-25T08:46:26.206 INFO:tasks.rados.rados.0.trial039.stdout:1075: finishing write tid 1 to trial03942481-23 2026-02-25T08:46:26.206 INFO:tasks.rados.rados.0.trial039.stdout:1077: finishing write tid 3 to trial03942481-2 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:1077: finishing write tid 4 to trial03942481-2 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:1077: oid 2 updating version 0 to 207 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:1077: oid 2 version 207 is already newer than 207 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:1077: oid 2 version 207 is already newer than 207 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 2 v 207 (ObjNum 508 snap 141 seq_num 508) dirty exists 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:1077: left oid 2 (ObjNum 508 snap 141 seq_num 508) 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:1079: finishing write tid 1 to trial03942481-49 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:1075: finishing write tid 2 to trial03942481-23 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:1075: finishing write tid 3 to trial03942481-23 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:1075: finishing write tid 4 to trial03942481-23 2026-02-25T08:46:26.207 INFO:tasks.rados.rados.0.trial039.stdout:1075: oid 23 updating version 0 to 239 2026-02-25T08:46:26.208 INFO:tasks.rados.rados.0.trial039.stdout:1075: oid 23 version 239 is already newer than 239 2026-02-25T08:46:26.208 INFO:tasks.rados.rados.0.trial039.stdout:1075: oid 23 version 239 is already newer than 237 2026-02-25T08:46:26.208 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 23 v 239 (ObjNum 507 snap 141 seq_num 507) dirty exists 2026-02-25T08:46:26.208 INFO:tasks.rados.rados.0.trial039.stdout:1075: left oid 23 (ObjNum 507 snap 141 seq_num 507) 2026-02-25T08:46:26.208 INFO:tasks.rados.rados.0.trial039.stdout:1079: finishing write tid 2 to trial03942481-49 2026-02-25T08:46:26.208 INFO:tasks.rados.rados.0.trial039.stdout:1078: finishing write tid 1 to trial03942481-5 2026-02-25T08:46:26.208 INFO:tasks.rados.rados.0.trial039.stdout:1079: finishing write tid 3 to trial03942481-49 2026-02-25T08:46:26.208 INFO:tasks.rados.rados.0.trial039.stdout:1079: finishing write tid 4 to trial03942481-49 2026-02-25T08:46:26.208 INFO:tasks.rados.rados.0.trial039.stdout:1079: oid 49 updating version 0 to 123 2026-02-25T08:46:26.208 INFO:tasks.rados.rados.0.trial039.stdout:1079: oid 49 version 123 is already newer than 123 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:1079: oid 49 version 123 is already newer than 121 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 49 v 123 (ObjNum 510 snap 141 seq_num 510) dirty exists 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:1079: left oid 49 (ObjNum 510 snap 141 seq_num 510) 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:1078: finishing write tid 2 to trial03942481-5 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:1078: finishing write tid 3 to trial03942481-5 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:1078: finishing write tid 4 to trial03942481-5 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:1078: oid 5 updating version 0 to 243 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:1078: oid 5 version 243 is already newer than 243 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:1078: oid 5 version 243 is already newer than 243 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 5 v 243 (ObjNum 509 snap 141 seq_num 509) dirty exists 2026-02-25T08:46:26.209 INFO:tasks.rados.rados.0.trial039.stdout:1078: left oid 5 (ObjNum 509 snap 141 seq_num 509) 2026-02-25T08:46:26.221 INFO:tasks.rados.rados.0.trial039.stdout:1075: done (5 left) 2026-02-25T08:46:26.221 INFO:tasks.rados.rados.0.trial039.stdout:1076: done (4 left) 2026-02-25T08:46:26.221 INFO:tasks.rados.rados.0.trial039.stdout:1077: done (3 left) 2026-02-25T08:46:26.221 INFO:tasks.rados.rados.0.trial039.stdout:1078: done (2 left) 2026-02-25T08:46:26.221 INFO:tasks.rados.rados.0.trial039.stdout:1079: done (1 left) 2026-02-25T08:46:26.221 INFO:tasks.rados.rados.0.trial039.stdout:1081: done (0 left) 2026-02-25T08:46:26.221 INFO:tasks.rados.rados.0.trial039.stdout:1082: read oid 19 snap -1 2026-02-25T08:46:26.222 INFO:tasks.rados.rados.0.trial039.stdout:1082: expect (ObjNum 500 snap 139 seq_num 500) 2026-02-25T08:46:26.222 INFO:tasks.rados.rados.0.trial039.stdout:1083: read oid 30 snap -1 2026-02-25T08:46:26.222 INFO:tasks.rados.rados.0.trial039.stdout:1083: expect deleted 2026-02-25T08:46:26.222 INFO:tasks.rados.rados.0.trial039.stdout:1084: snap_create 2026-02-25T08:46:26.240 INFO:tasks.rados.rados.0.trial039.stdout:1082: expect (ObjNum 500 snap 139 seq_num 500) 2026-02-25T08:46:26.240 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:26.295 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2350983 2026-02-25T08:46:27.042 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~292s 2026-02-25T08:46:27.209 INFO:tasks.rados.rados.0.trial039.stdout:1082: done (2 left) 2026-02-25T08:46:27.209 INFO:tasks.rados.rados.0.trial039.stdout:1083: done (1 left) 2026-02-25T08:46:27.209 INFO:tasks.rados.rados.0.trial039.stdout:1084: done (0 left) 2026-02-25T08:46:27.210 INFO:tasks.rados.rados.0.trial039.stdout:1085: snap_create 2026-02-25T08:46:27.567 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:27.644 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:27.641+0000 7fe58a6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:27.647 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:27.645+0000 7fe58a6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:27.674 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:27.691 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:27.691 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:28.200 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:28.210 INFO:tasks.rados.rados.0.trial039.stdout:1085: done (0 left) 2026-02-25T08:46:28.210 INFO:tasks.rados.rados.0.trial039.stdout:1086: write (excl) oid 31 current snap is 143 2026-02-25T08:46:28.210 INFO:tasks.rados.rados.0.trial039.stdout:1086: seq_num 512 ranges {605184=723139,1877957=727227} 2026-02-25T08:46:28.215 INFO:tasks.rados.rados.0.trial039.stdout:1086: writing trial03942481-31 from 605184 to 1328323 tid 1 2026-02-25T08:46:28.220 INFO:tasks.rados.rados.0.trial039.stdout:1086: writing trial03942481-31 from 1877957 to 2605184 tid 2 2026-02-25T08:46:28.220 INFO:tasks.rados.rados.0.trial039.stdout:1087: read oid 38 snap -1 2026-02-25T08:46:28.221 INFO:tasks.rados.rados.0.trial039.stdout:1087: expect deleted 2026-02-25T08:46:28.221 INFO:tasks.rados.rados.0.trial039.stdout:1088: write (excl) oid 27 current snap is 143 2026-02-25T08:46:28.221 INFO:tasks.rados.rados.0.trial039.stdout:1088: seq_num 513 ranges {621991=798388,1936439=639509} 2026-02-25T08:46:28.225 INFO:tasks.rados.rados.0.trial039.stdout:1088: writing trial03942481-27 from 621991 to 1420379 tid 1 2026-02-25T08:46:28.230 INFO:tasks.rados.rados.0.trial039.stdout:1088: writing trial03942481-27 from 1936439 to 2575948 tid 2 2026-02-25T08:46:28.246 INFO:tasks.rados.rados.0.trial039.stdout:1089: snap_remove snap 126 2026-02-25T08:46:28.276 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:28.279 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:29.215 INFO:tasks.rados.rados.0.trial039.stdout:1089: done (3 left) 2026-02-25T08:46:29.215 INFO:tasks.rados.rados.0.trial039.stdout:1090: write oid 47 current snap is 143 2026-02-25T08:46:29.215 INFO:tasks.rados.rados.0.trial039.stdout:1090: seq_num 514 ranges {638798=473637,1594921=783167} 2026-02-25T08:46:29.219 INFO:tasks.rados.rados.0.trial039.stdout:1090: writing trial03942481-47 from 638798 to 1112435 tid 1 2026-02-25T08:46:29.225 INFO:tasks.rados.rados.0.trial039.stdout:1090: writing trial03942481-47 from 1594921 to 2378088 tid 2 2026-02-25T08:46:29.225 INFO:tasks.rados.rados.0.trial039.stdout:1091: read oid 46 snap -1 2026-02-25T08:46:29.225 INFO:tasks.rados.rados.0.trial039.stdout:1091: expect (ObjNum 499 snap 139 seq_num 499) 2026-02-25T08:46:29.225 INFO:tasks.rados.rados.0.trial039.stdout:1092: write oid 23 current snap is 143 2026-02-25T08:46:29.225 INFO:tasks.rados.rados.0.trial039.stdout:1092: seq_num 515 ranges {655605=548886,1653403=643178} 2026-02-25T08:46:29.230 INFO:tasks.rados.rados.0.trial039.stdout:1092: writing trial03942481-23 from 655605 to 1204491 tid 1 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:1092: writing trial03942481-23 from 1653403 to 2296581 tid 2 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:1093: delete oid 15 current snap is 143 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:1088: finishing write tid 1 to trial03942481-27 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:1088: finishing write tid 2 to trial03942481-27 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:1088: finishing write tid 3 to trial03942481-27 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:1088: finishing write tid 4 to trial03942481-27 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:1088: oid 27 updating version 0 to 127 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:1088: oid 27 version 127 is already newer than 125 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:1088: oid 27 version 127 is already newer than 127 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 27 v 127 (ObjNum 513 snap 143 seq_num 513) dirty exists 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial039.stdout:1088: left oid 27 (ObjNum 513 snap 143 seq_num 513) 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial039.stdout:1086: finishing write tid 1 to trial03942481-31 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial039.stdout:1086: finishing write tid 2 to trial03942481-31 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial039.stdout:1086: finishing write tid 3 to trial03942481-31 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial039.stdout:1086: finishing write tid 4 to trial03942481-31 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial039.stdout:1086: oid 31 updating version 0 to 405 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial039.stdout:1086: oid 31 version 405 is already newer than 405 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial039.stdout:1086: oid 31 version 405 is already newer than 403 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 31 v 405 (ObjNum 512 snap 143 seq_num 512) dirty exists 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial039.stdout:1086: left oid 31 (ObjNum 512 snap 143 seq_num 512) 2026-02-25T08:46:29.281 INFO:tasks.rados.rados.0.trial039.stdout:1091: expect (ObjNum 499 snap 139 seq_num 499) 2026-02-25T08:46:29.353 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2380691 2026-02-25T08:46:29.353 INFO:tasks.rados.rados.0.trial039.stdout:1090: finishing write tid 1 to trial03942481-47 2026-02-25T08:46:29.353 INFO:tasks.rados.rados.0.trial039.stdout:1086: done (6 left) 2026-02-25T08:46:29.353 INFO:tasks.rados.rados.0.trial039.stdout:1087: done (5 left) 2026-02-25T08:46:29.353 INFO:tasks.rados.rados.0.trial039.stdout:1088: done (4 left) 2026-02-25T08:46:29.353 INFO:tasks.rados.rados.0.trial039.stdout:1091: done (3 left) 2026-02-25T08:46:29.353 INFO:tasks.rados.rados.0.trial039.stdout:1093: done (2 left) 2026-02-25T08:46:29.354 INFO:tasks.rados.rados.0.trial039.stdout:1094: read oid 1 snap -1 2026-02-25T08:46:29.354 INFO:tasks.rados.rados.0.trial039.stdout:1094: expect deleted 2026-02-25T08:46:29.354 INFO:tasks.rados.rados.0.trial039.stdout:1095: snap_remove snap 141 2026-02-25T08:46:29.693 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:29.769 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:29.766+0000 7fb36865a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:29.772 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:29.769+0000 7fb36865a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:29.799 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:29.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:29.817 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:30.216 INFO:tasks.rados.rados.0.trial039.stdout:1095: done (3 left) 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:1096: snap_create 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:1090: finishing write tid 2 to trial03942481-47 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:1090: finishing write tid 3 to trial03942481-47 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:1090: finishing write tid 4 to trial03942481-47 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:1090: oid 47 updating version 0 to 170 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:1090: oid 47 version 170 is already newer than 170 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:1090: oid 47 version 170 is already newer than 170 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 47 v 170 (ObjNum 514 snap 143 seq_num 514) dirty exists 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:1090: left oid 47 (ObjNum 514 snap 143 seq_num 514) 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:1092: finishing write tid 1 to trial03942481-23 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial039.stdout:1092: finishing write tid 2 to trial03942481-23 2026-02-25T08:46:30.218 INFO:tasks.rados.rados.0.trial039.stdout:1092: finishing write tid 3 to trial03942481-23 2026-02-25T08:46:30.218 INFO:tasks.rados.rados.0.trial039.stdout:1092: finishing write tid 4 to trial03942481-23 2026-02-25T08:46:30.218 INFO:tasks.rados.rados.0.trial039.stdout:1092: oid 23 updating version 0 to 253 2026-02-25T08:46:30.218 INFO:tasks.rados.rados.0.trial039.stdout:1092: oid 23 version 253 is already newer than 253 2026-02-25T08:46:30.218 INFO:tasks.rados.rados.0.trial039.stdout:1092: oid 23 version 253 is already newer than 253 2026-02-25T08:46:30.218 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 23 v 253 (ObjNum 515 snap 143 seq_num 515) dirty exists 2026-02-25T08:46:30.218 INFO:tasks.rados.rados.0.trial039.stdout:1092: left oid 23 (ObjNum 515 snap 143 seq_num 515) 2026-02-25T08:46:31.218 INFO:tasks.rados.rados.0.trial039.stdout:1090: done (3 left) 2026-02-25T08:46:31.219 INFO:tasks.rados.rados.0.trial039.stdout:1092: done (2 left) 2026-02-25T08:46:31.219 INFO:tasks.rados.rados.0.trial039.stdout:1094: done (1 left) 2026-02-25T08:46:31.219 INFO:tasks.rados.rados.0.trial039.stdout:1096: done (0 left) 2026-02-25T08:46:31.219 INFO:tasks.rados.rados.0.trial039.stdout:1097: write oid 26 current snap is 144 2026-02-25T08:46:31.219 INFO:tasks.rados.rados.0.trial039.stdout:1097: seq_num 517 ranges {689219=415737,1770367=646847} 2026-02-25T08:46:31.223 INFO:tasks.rados.rados.0.trial039.stdout:1097: writing trial03942481-26 from 689219 to 1104956 tid 1 2026-02-25T08:46:31.228 INFO:tasks.rados.rados.0.trial039.stdout:1097: writing trial03942481-26 from 1770367 to 2417214 tid 2 2026-02-25T08:46:31.229 INFO:tasks.rados.rados.0.trial039.stdout:1098: write (excl) oid 42 current snap is 144 2026-02-25T08:46:31.229 INFO:tasks.rados.rados.0.trial039.stdout:1098: seq_num 518 ranges {706026=490986,1828849=790505} 2026-02-25T08:46:31.233 INFO:tasks.rados.rados.0.trial039.stdout:1098: writing trial03942481-42 from 706026 to 1197012 tid 1 2026-02-25T08:46:31.239 INFO:tasks.rados.rados.0.trial039.stdout:1098: writing trial03942481-42 from 1828849 to 2619354 tid 2 2026-02-25T08:46:31.239 INFO:tasks.rados.rados.0.trial039.stdout:1099: read oid 35 snap -1 2026-02-25T08:46:31.239 INFO:tasks.rados.rados.0.trial039.stdout:1099: expect (ObjNum 486 snap 133 seq_num 486) 2026-02-25T08:46:31.239 INFO:tasks.rados.rados.0.trial039.stdout:1100: read oid 30 snap -1 2026-02-25T08:46:31.239 INFO:tasks.rados.rados.0.trial039.stdout:1100: expect deleted 2026-02-25T08:46:31.240 INFO:tasks.rados.rados.0.trial039.stdout:1101: read oid 16 snap -1 2026-02-25T08:46:31.240 INFO:tasks.rados.rados.0.trial039.stdout:1101: expect deleted 2026-02-25T08:46:31.240 INFO:tasks.rados.rados.0.trial039.stdout:1102: write oid 39 current snap is 144 2026-02-25T08:46:31.240 INFO:tasks.rados.rados.0.trial039.stdout:1102: seq_num 519 ranges {722833=566235,1887331=650516} 2026-02-25T08:46:31.243 INFO:tasks.rados.rados.0.trial039.stdout:1102: writing trial03942481-39 from 722833 to 1289068 tid 1 2026-02-25T08:46:31.248 INFO:tasks.rados.rados.0.trial039.stdout:1102: writing trial03942481-39 from 1887331 to 2537847 tid 2 2026-02-25T08:46:31.248 INFO:tasks.rados.rados.0.trial039.stdout:1103: read oid 40 snap -1 2026-02-25T08:46:31.249 INFO:tasks.rados.rados.0.trial039.stdout:1103: expect (ObjNum 452 snap 119 seq_num 452) 2026-02-25T08:46:31.249 INFO:tasks.rados.rados.0.trial039.stdout:1104: snap_create 2026-02-25T08:46:31.262 INFO:tasks.rados.rados.0.trial039.stdout:1099: expect (ObjNum 486 snap 133 seq_num 486) 2026-02-25T08:46:31.291 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:31.319 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:31.348 INFO:tasks.rados.rados.0.trial039.stdout:1097: finishing write tid 1 to trial03942481-26 2026-02-25T08:46:31.348 INFO:tasks.rados.rados.0.trial039.stdout:1103: expect (ObjNum 452 snap 119 seq_num 452) 2026-02-25T08:46:31.348 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 68966 2026-02-25T08:46:31.433 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2526100 2026-02-25T08:46:31.481 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2380533 2026-02-25T08:46:31.507 INFO:tasks.rados.rados.0.trial039.stdout:1098: finishing write tid 1 to trial03942481-42 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:1097: finishing write tid 2 to trial03942481-26 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:1097: finishing write tid 3 to trial03942481-26 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:1097: finishing write tid 4 to trial03942481-26 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:1097: oid 26 updating version 0 to 95 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:1097: oid 26 version 95 is already newer than 95 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:1097: oid 26 version 95 is already newer than 95 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 26 v 95 (ObjNum 517 snap 144 seq_num 517) dirty exists 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:1097: left oid 26 (ObjNum 517 snap 144 seq_num 517) 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:1098: finishing write tid 2 to trial03942481-42 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:1098: finishing write tid 3 to trial03942481-42 2026-02-25T08:46:31.508 INFO:tasks.rados.rados.0.trial039.stdout:1098: finishing write tid 4 to trial03942481-42 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1098: oid 42 updating version 0 to 413 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1098: oid 42 version 413 is already newer than 413 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1098: oid 42 version 413 is already newer than 411 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 42 v 413 (ObjNum 518 snap 144 seq_num 518) dirty exists 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1098: left oid 42 (ObjNum 518 snap 144 seq_num 518) 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1102: finishing write tid 1 to trial03942481-39 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1102: finishing write tid 2 to trial03942481-39 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1102: finishing write tid 3 to trial03942481-39 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1102: finishing write tid 4 to trial03942481-39 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1102: oid 39 updating version 0 to 417 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1102: oid 39 version 417 is already newer than 417 2026-02-25T08:46:31.509 INFO:tasks.rados.rados.0.trial039.stdout:1102: oid 39 version 417 is already newer than 417 2026-02-25T08:46:31.510 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 39 v 417 (ObjNum 519 snap 144 seq_num 519) dirty exists 2026-02-25T08:46:31.510 INFO:tasks.rados.rados.0.trial039.stdout:1102: left oid 39 (ObjNum 519 snap 144 seq_num 519) 2026-02-25T08:46:31.819 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:31.898 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:31.894+0000 7f6c11d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:31.900 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:31.898+0000 7f6c11d5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:31.928 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:31.945 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:32.227 INFO:tasks.rados.rados.0.trial039.stdout:1097: done (7 left) 2026-02-25T08:46:32.227 INFO:tasks.rados.rados.0.trial039.stdout:1098: done (6 left) 2026-02-25T08:46:32.228 INFO:tasks.rados.rados.0.trial039.stdout:1099: done (5 left) 2026-02-25T08:46:32.228 INFO:tasks.rados.rados.0.trial039.stdout:1100: done (4 left) 2026-02-25T08:46:32.228 INFO:tasks.rados.rados.0.trial039.stdout:1101: done (3 left) 2026-02-25T08:46:32.228 INFO:tasks.rados.rados.0.trial039.stdout:1102: done (2 left) 2026-02-25T08:46:32.228 INFO:tasks.rados.rados.0.trial039.stdout:1103: done (1 left) 2026-02-25T08:46:32.228 INFO:tasks.rados.rados.0.trial039.stdout:1104: done (0 left) 2026-02-25T08:46:32.228 INFO:tasks.rados.rados.0.trial039.stdout:1105: write oid 42 current snap is 145 2026-02-25T08:46:32.228 INFO:tasks.rados.rados.0.trial039.stdout:1105: seq_num 520 ranges {739640=641484,1829460=794174} 2026-02-25T08:46:32.232 INFO:tasks.rados.rados.0.trial039.stdout:1105: writing trial03942481-42 from 739640 to 1381124 tid 1 2026-02-25T08:46:32.238 INFO:tasks.rados.rados.0.trial039.stdout:1105: writing trial03942481-42 from 1829460 to 2623634 tid 2 2026-02-25T08:46:32.238 INFO:tasks.rados.rados.0.trial039.stdout:1106: snap_remove snap 129 2026-02-25T08:46:32.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~297s 2026-02-25T08:46:33.101 INFO:tasks.rados.rados.0.trial039.stdout:1106: done (1 left) 2026-02-25T08:46:33.102 INFO:tasks.rados.rados.0.trial039.stdout:1107: read oid 39 snap -1 2026-02-25T08:46:33.102 INFO:tasks.rados.rados.0.trial039.stdout:1107: expect (ObjNum 519 snap 144 seq_num 519) 2026-02-25T08:46:33.102 INFO:tasks.rados.rados.0.trial039.stdout:1108: snap_create 2026-02-25T08:46:33.102 INFO:tasks.rados.rados.0.trial039.stdout:1105: finishing write tid 1 to trial03942481-42 2026-02-25T08:46:33.102 INFO:tasks.rados.rados.0.trial039.stdout:1105: finishing write tid 2 to trial03942481-42 2026-02-25T08:46:33.102 INFO:tasks.rados.rados.0.trial039.stdout:1105: finishing write tid 3 to trial03942481-42 2026-02-25T08:46:33.102 INFO:tasks.rados.rados.0.trial039.stdout:1105: finishing write tid 4 to trial03942481-42 2026-02-25T08:46:33.102 INFO:tasks.rados.rados.0.trial039.stdout:1105: oid 42 updating version 0 to 423 2026-02-25T08:46:33.102 INFO:tasks.rados.rados.0.trial039.stdout:1105: oid 42 version 423 is already newer than 423 2026-02-25T08:46:33.102 INFO:tasks.rados.rados.0.trial039.stdout:1105: oid 42 version 423 is already newer than 423 2026-02-25T08:46:33.103 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 42 v 423 (ObjNum 520 snap 145 seq_num 520) dirty exists 2026-02-25T08:46:33.103 INFO:tasks.rados.rados.0.trial039.stdout:1105: left oid 42 (ObjNum 520 snap 145 seq_num 520) 2026-02-25T08:46:33.129 INFO:tasks.rados.rados.0.trial039.stdout:1107: expect (ObjNum 519 snap 144 seq_num 519) 2026-02-25T08:46:33.165 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:33.221 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2423253 2026-02-25T08:46:33.281 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:33.355 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:33.358 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:33.358 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:33.946 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:34.023 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:34.019+0000 7fba500cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:34.025 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:34.023+0000 7fba500cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:34.053 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:34.070 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:34.099 INFO:tasks.rados.rados.0.trial039.stdout:1105: done (2 left) 2026-02-25T08:46:34.100 INFO:tasks.rados.rados.0.trial039.stdout:1107: done (1 left) 2026-02-25T08:46:34.100 INFO:tasks.rados.rados.0.trial039.stdout:1108: done (0 left) 2026-02-25T08:46:34.100 INFO:tasks.rados.rados.0.trial039.stdout:1109: snap_create 2026-02-25T08:46:35.102 INFO:tasks.rados.rados.0.trial039.stdout:1109: done (0 left) 2026-02-25T08:46:35.102 INFO:tasks.rados.rados.0.trial039.stdout:1110: snap_remove snap 145 2026-02-25T08:46:36.071 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:36.101 INFO:tasks.rados.rados.0.trial039.stdout:1110: done (0 left) 2026-02-25T08:46:36.101 INFO:tasks.rados.rados.0.trial039.stdout:1111: delete oid 13 current snap is 147 2026-02-25T08:46:36.122 INFO:tasks.rados.rados.0.trial039.stdout:1111: done (0 left) 2026-02-25T08:46:36.122 INFO:tasks.rados.rados.0.trial039.stdout:1112: write (excl) oid 10 current snap is 147 2026-02-25T08:46:36.122 INFO:tasks.rados.rados.0.trial039.stdout:1112: seq_num 522 ranges {773254=791982,2346424=426830} 2026-02-25T08:46:36.128 INFO:tasks.rados.rados.0.trial039.stdout:1112: writing trial03942481-10 from 773254 to 1565236 tid 1 2026-02-25T08:46:36.133 INFO:tasks.rados.rados.0.trial039.stdout:1112: writing trial03942481-10 from 2346424 to 2773254 tid 2 2026-02-25T08:46:36.133 INFO:tasks.rados.rados.0.trial039.stdout:1113: write (excl) oid 1 current snap is 147 2026-02-25T08:46:36.133 INFO:tasks.rados.rados.0.trial039.stdout:1113: seq_num 523 ranges {790061=467231,2004906=541501} 2026-02-25T08:46:36.138 INFO:tasks.rados.rados.0.trial039.stdout:1113: writing trial03942481-1 from 790061 to 1257292 tid 1 2026-02-25T08:46:36.143 INFO:tasks.rados.rados.0.trial039.stdout:1113: writing trial03942481-1 from 2004906 to 2546407 tid 2 2026-02-25T08:46:36.144 INFO:tasks.rados.rados.0.trial039.stdout:1114: read oid 29 snap -1 2026-02-25T08:46:36.144 INFO:tasks.rados.rados.0.trial039.stdout:1114: expect deleted 2026-02-25T08:46:36.144 INFO:tasks.rados.rados.0.trial039.stdout:1115: read oid 26 snap -1 2026-02-25T08:46:36.144 INFO:tasks.rados.rados.0.trial039.stdout:1115: expect (ObjNum 517 snap 144 seq_num 517) 2026-02-25T08:46:36.144 INFO:tasks.rados.rados.0.trial039.stdout:1116: snap_remove snap 146 2026-02-25T08:46:36.153 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:36.149+0000 7f541e9dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:36.156 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:36.154+0000 7f541e9dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:36.199 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:36.216 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:37.108 INFO:tasks.rados.rados.0.trial039.stdout:1116: done (4 left) 2026-02-25T08:46:37.108 INFO:tasks.rados.rados.0.trial039.stdout:1117: write (excl) oid 14 current snap is 147 2026-02-25T08:46:37.108 INFO:tasks.rados.rados.0.trial039.stdout:1112: finishing write tid 1 to trial03942481-10 2026-02-25T08:46:37.108 INFO:tasks.rados.rados.0.trial039.stdout:1117: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2026-02-25T08:46:37.111 INFO:tasks.rados.rados.0.trial039.stdout:1117: writing trial03942481-14 from 406868 to 949348 tid 1 2026-02-25T08:46:37.115 INFO:tasks.rados.rados.0.trial039.stdout:1117: writing trial03942481-14 from 1547035 to 1948547 tid 2 2026-02-25T08:46:37.119 INFO:tasks.rados.rados.0.trial039.stdout:1117: writing trial03942481-14 from 2690354 to 2806868 tid 3 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:1118: read oid 12 snap -1 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:1118: expect (ObjNum 498 snap 139 seq_num 498) 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:1119: delete oid 44 current snap is 147 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:1113: finishing write tid 1 to trial03942481-1 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:1112: finishing write tid 2 to trial03942481-10 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:1112: finishing write tid 3 to trial03942481-10 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:1112: finishing write tid 4 to trial03942481-10 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:1112: oid 10 updating version 0 to 173 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:1112: oid 10 version 173 is already newer than 173 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:1112: oid 10 version 173 is already newer than 171 2026-02-25T08:46:37.120 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 10 v 173 (ObjNum 522 snap 147 seq_num 522) dirty exists 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:1112: left oid 10 (ObjNum 522 snap 147 seq_num 522) 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:1113: finishing write tid 2 to trial03942481-1 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:1113: finishing write tid 3 to trial03942481-1 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:1113: finishing write tid 4 to trial03942481-1 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:1113: oid 1 updating version 0 to 98 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:1113: oid 1 version 98 is already newer than 98 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:1113: oid 1 version 98 is already newer than 96 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 1 v 98 (ObjNum 523 snap 147 seq_num 523) dirty exists 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:1113: left oid 1 (ObjNum 523 snap 147 seq_num 523) 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:1115: expect (ObjNum 517 snap 144 seq_num 517) 2026-02-25T08:46:37.121 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 0 2026-02-25T08:46:37.174 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 2290914 2026-02-25T08:46:37.177 INFO:tasks.rados.rados.0.trial039.stdout:1118: expect (ObjNum 498 snap 139 seq_num 498) 2026-02-25T08:46:37.178 INFO:tasks.rados.rados.0.trial039.stdout:seeking to 604962 2026-02-25T08:46:37.219 INFO:tasks.rados.rados.0.trial039.stdout:1117: finishing write tid 1 to trial03942481-14 2026-02-25T08:46:37.219 INFO:tasks.rados.rados.0.trial039.stdout:1117: finishing write tid 2 to trial03942481-14 2026-02-25T08:46:37.219 INFO:tasks.rados.rados.0.trial039.stdout:1112: done (6 left) 2026-02-25T08:46:37.219 INFO:tasks.rados.rados.0.trial039.stdout:1113: done (5 left) 2026-02-25T08:46:37.219 INFO:tasks.rados.rados.0.trial039.stdout:1114: done (4 left) 2026-02-25T08:46:37.219 INFO:tasks.rados.rados.0.trial039.stdout:1115: done (3 left) 2026-02-25T08:46:37.219 INFO:tasks.rados.rados.0.trial039.stdout:1118: done (2 left) 2026-02-25T08:46:37.219 INFO:tasks.rados.rados.0.trial039.stdout:1119: done (1 left) 2026-02-25T08:46:37.220 INFO:tasks.rados.rados.0.trial039.stdout:1120: write oid 42 current snap is 147 2026-02-25T08:46:37.220 INFO:tasks.rados.rados.0.trial039.stdout:1117: finishing write tid 3 to trial03942481-14 2026-02-25T08:46:37.220 INFO:tasks.rados.rados.0.trial039.stdout:1117: finishing write tid 4 to trial03942481-14 2026-02-25T08:46:37.220 INFO:tasks.rados.rados.0.trial039.stdout:1120: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2026-02-25T08:46:37.222 INFO:tasks.rados.rados.0.trial039.stdout:1120: writing trial03942481-42 from 440482 to 849813 tid 1 2026-02-25T08:46:37.226 INFO:tasks.rados.rados.0.trial039.stdout:1120: writing trial03942481-42 from 1380352 to 1785533 tid 2 2026-02-25T08:46:37.230 INFO:tasks.rados.rados.0.trial039.stdout:1120: writing trial03942481-42 from 2461553 to 2840482 tid 3 2026-02-25T08:46:37.231 INFO:tasks.rados.rados.0.trial039.stdout:1121: read oid 44 snap -1 2026-02-25T08:46:37.231 INFO:tasks.rados.rados.0.trial039.stdout:1121: expect deleted 2026-02-25T08:46:37.231 INFO:tasks.rados.rados.0.trial039.stdout:1122: delete oid 30 current snap is 147 2026-02-25T08:46:37.231 INFO:tasks.rados.rados.0.trial039.stdout:1117: finishing write tid 5 to trial03942481-14 2026-02-25T08:46:37.231 INFO:tasks.rados.rados.0.trial039.stdout:1117: oid 14 updating version 0 to 223 2026-02-25T08:46:37.231 INFO:tasks.rados.rados.0.trial039.stdout:1117: oid 14 version 223 is already newer than 223 2026-02-25T08:46:37.231 INFO:tasks.rados.rados.0.trial039.stdout:1117: oid 14 version 223 is already newer than 223 2026-02-25T08:46:37.231 INFO:tasks.rados.rados.0.trial039.stdout:1117: oid 14 version 223 is already newer than 220 2026-02-25T08:46:37.231 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 14 v 223 (ObjNum 524 snap 147 seq_num 524) dirty exists 2026-02-25T08:46:37.231 INFO:tasks.rados.rados.0.trial039.stdout:1117: left oid 14 (ObjNum 524 snap 147 seq_num 524) 2026-02-25T08:46:37.233 INFO:tasks.rados.rados.0.trial039.stdout:1117: done (3 left) 2026-02-25T08:46:37.233 INFO:tasks.rados.rados.0.trial039.stdout:1122: done (2 left) 2026-02-25T08:46:37.233 INFO:tasks.rados.rados.0.trial039.stdout:1123: read oid 29 snap -1 2026-02-25T08:46:37.233 INFO:tasks.rados.rados.0.trial039.stdout:1123: expect deleted 2026-02-25T08:46:37.233 INFO:tasks.rados.rados.0.trial039.stdout:1124: snap_create 2026-02-25T08:46:37.249 INFO:tasks.rados.rados.0.trial039.stdout:1120: finishing write tid 1 to trial03942481-42 2026-02-25T08:46:37.258 INFO:tasks.rados.rados.0.trial039.stdout:1120: finishing write tid 2 to trial03942481-42 2026-02-25T08:46:37.265 INFO:tasks.rados.rados.0.trial039.stdout:1120: finishing write tid 3 to trial03942481-42 2026-02-25T08:46:37.276 INFO:tasks.rados.rados.0.trial039.stdout:1120: finishing write tid 4 to trial03942481-42 2026-02-25T08:46:37.277 INFO:tasks.rados.rados.0.trial039.stdout:1120: finishing write tid 5 to trial03942481-42 2026-02-25T08:46:37.277 INFO:tasks.rados.rados.0.trial039.stdout:1120: oid 42 updating version 0 to 435 2026-02-25T08:46:37.277 INFO:tasks.rados.rados.0.trial039.stdout:1120: oid 42 version 435 is already newer than 435 2026-02-25T08:46:37.277 INFO:tasks.rados.rados.0.trial039.stdout:1120: oid 42 version 435 is already newer than 435 2026-02-25T08:46:37.277 INFO:tasks.rados.rados.0.trial039.stdout:1120: oid 42 version 435 is already newer than 435 2026-02-25T08:46:37.277 INFO:tasks.rados.rados.0.trial039.stdout:update_object_version oid 42 v 435 (ObjNum 526 snap 147 seq_num 526) dirty exists 2026-02-25T08:46:37.277 INFO:tasks.rados.rados.0.trial039.stdout:1120: left oid 42 (ObjNum 526 snap 147 seq_num 526) 2026-02-25T08:46:38.050 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~303s 2026-02-25T08:46:38.050 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2026-02-25T08:46:38.108 INFO:tasks.rados.rados.0.trial039.stdout:1120: done (3 left) 2026-02-25T08:46:38.108 INFO:tasks.rados.rados.0.trial039.stdout:1121: done (2 left) 2026-02-25T08:46:38.108 INFO:tasks.rados.rados.0.trial039.stdout:1123: done (1 left) 2026-02-25T08:46:38.109 INFO:tasks.rados.rados.0.trial039.stdout:1124: done (0 left) 2026-02-25T08:46:38.109 INFO:tasks.rados.rados.0.trial039.stdout:1125: write (excl) oid 26 current snap is 148 2026-02-25T08:46:38.109 INFO:tasks.rados.rados.0.trial039.stdout:1125: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2026-02-25T08:46:38.112 INFO:tasks.rados.rados.0.trial039.stdout:1125: writing trial03942481-26 from 474096 to 1033925 tid 1 2026-02-25T08:46:38.116 INFO:tasks.rados.rados.0.trial039.stdout:1125: writing trial03942481-26 from 1780963 to 2189813 tid 2 2026-02-25T08:46:38.119 INFO:tasks.rados.rados.0.trial039.stdout:1125: writing trial03942481-26 from 2800046 to 2874096 tid 3 2026-02-25T08:46:38.119 INFO:tasks.rados.rados.0.trial039.stdout:1126: snap_remove snap 138 2026-02-25T08:46:38.217 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:38.295 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:38.291+0000 7efce4c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:38.297 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:46:38.295+0000 7efce4c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:38.324 INFO:teuthology.orchestra.run.trial039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-25T08:46:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:38.341 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:38.359 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:38.431 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:38.434 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:38.434 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:38.554 INFO:tasks.ceph.osd.0:Sent signal 15 2026-02-25T08:46:38.555 INFO:tasks.ceph.osd.1:Sent signal 15 2026-02-25T08:46:38.555 INFO:tasks.ceph.osd.3:Sent signal 15 2026-02-25T08:46:38.555 INFO:tasks.ceph.mon.a:Sent signal 15 2026-02-25T08:46:38.556 INFO:tasks.ceph.mgr.x:Sent signal 15 2026-02-25T08:46:38.556 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing thrasher OSDThrasher 2026-02-25T08:46:38.556 INFO:tasks.ceph.mon.a.trial039.stderr:2026-02-25T08:46:38.554+0000 7f2b817fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 40198) UID: 0 2026-02-25T08:46:38.557 INFO:tasks.ceph.mon.a.trial039.stderr:2026-02-25T08:46:38.554+0000 7f2b817fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2026-02-25T08:46:38.557 INFO:tasks.ceph.mgr.x.trial091.stderr:2026-02-25T08:46:38.556+0000 7fcf313f8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 39310) UID: 0 2026-02-25T08:46:38.557 INFO:tasks.ceph.mgr.x.trial091.stderr:2026-02-25T08:46:38.556+0000 7fcf313f8640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2026-02-25T08:46:43.435 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:43.509 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:43.513 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:48.514 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:48.586 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:48.590 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:53.591 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:53.663 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:53.666 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:46:58.667 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:46:58.739 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:58.742 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:47:03.743 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:47:03.814 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:03.817 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:03.818 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:47:08.819 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:47:08.891 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:08.894 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:47:13.896 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:47:13.967 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:13.970 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:47:18.972 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:47:19.042 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:19.045 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:47:24.045 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:47:24.117 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:24.120 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:24.120 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:47:29.121 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:47:29.193 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:29.196 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-25T08:47:34.197 DEBUG:teuthology.orchestra.run.trial091:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-25T08:47:34.267 INFO:teuthology.orchestra.run.trial091.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:34.275 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.2 restart 2026-02-25T08:47:34.275 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1384, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.2 restart 2026-02-25T08:47:34.276 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-25T08:47:34.277 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing remote process ceph-test-rados-ceph 2026-02-25T08:47:34.277 DEBUG:tasks.rados:Stopping ceph-test-rados-ceph due to exception The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T08:47:34.277 INFO:tasks.rados:Stopping instance 0 2026-02-25T08:47:34.288 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T08:47:34.289 ERROR:teuthology.run_tasks:Manager failed: rados Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T08:47:34.290 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2026-02-25T08:47:34.301 INFO:tasks.thrashosds:joining thrashosds 2026-02-25T08:47:34.302 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-25T08:47:34.302 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T08:47:34.302 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:47:34.406 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:47:34.402+0000 7f2b55ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:49:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-25T08:49:34.361 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2970, in wait_for_all_osds_up while not self.are_all_osds_up(): ^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2960, in are_all_osds_up x = self.get_osd_dump() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2670, in get_osd_dump return self.get_osd_dump_json()['osds'] ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2662, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial039 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2026-02-25T08:49:34.367 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-25T08:49:34.384 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:49:34.384 DEBUG:teuthology.orchestra.run.trial039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:49:34.461 INFO:teuthology.orchestra.run.trial039.stderr:2026-02-25T08:49:34.457+0000 7f1219b83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:51:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-25T08:51:34.417 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial039 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-25T08:51:34.420 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-25T08:51:34.420 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-25T08:51:34.420 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-25T08:51:34.420 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:34.420 INFO:tasks.ceph.osd.0:Stopped 2026-02-25T08:51:34.420 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-25T08:51:34.420 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:34.421 INFO:tasks.ceph.osd.1:Stopped 2026-02-25T08:51:34.421 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-25T08:51:34.421 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:51:34.421 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 1525, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial039 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial091 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2026-02-25T08:51:34.423 INFO:tasks.ceph.osd.2:Stopped 2026-02-25T08:51:34.423 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-25T08:51:34.423 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:34.424 INFO:tasks.ceph.osd.3:Stopped 2026-02-25T08:51:34.424 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-25T08:51:34.424 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-25T08:51:34.424 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:34.424 INFO:tasks.ceph.mgr.x:Stopped 2026-02-25T08:51:34.424 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-25T08:51:34.424 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-25T08:51:34.424 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:51:34.424 INFO:tasks.ceph.mon.a:Stopped 2026-02-25T08:51:34.424 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-25T08:51:34.425 DEBUG:teuthology.orchestra.run.trial039:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v 'but it is still running' | grep -E -v 'objects unfound and apparently lost' | grep -E -v 'overall HEALTH_' | grep -E -v '\(OSDMAP_FLAGS\)' | grep -E -v '\(OSD_' | grep -E -v '\(PG_' | grep -E -v '\(POOL_' | grep -E -v '\(CACHE_POOL_' | grep -E -v '\(SMALLER_PGP_NUM\)' | grep -E -v '\(OBJECT_' | grep -E -v SLOW_OPS | grep -E -v '\(REQUEST_SLOW\)' | grep -E -v '\(TOO_FEW_PGS\)' | grep -E -v 'slow request' | grep -E -v 'timeout on replica' | grep -E -v 'late reservation from' | grep -E -v MON_DOWN | grep -E -v OSDMAP_FLAGS | grep -E -v OSD_DOWN | grep -E -v PG_DEGRADED | grep -E -v PG_AVAILABILITY | grep -E -v POOL_APP_NOT_ENABLED | grep -E -v 'mons down' | grep -E -v 'mon down' | grep -E -v 'out of quorum' | grep -E -v noscrub | grep -E -v nodeep-scrub | grep -E -v 'Degraded data redundancy' | grep -E -v 'is down' | grep -E -v 'osds down' | grep -E -v 'pg .*? is .*?degraded.*?, acting' | grep -E -v 'pg .*? is stuck' | grep -E -v 'pg degraded' | grep -E -v PG_BACKFILL_FULL | grep -E -v 'Low space hindering backfill .*? backfill_toofull' | grep -E -v OSD_ROOT_DOWN | grep -E -v 'pgs degraded' | grep -E -v 'pgs undersized' | grep -E -v 'is active.*backfill_toofull.*' | head -n 1 2026-02-25T08:51:34.453 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial039.front.sepia.ceph.com 2026-02-25T08:51:34.453 DEBUG:teuthology.orchestra.run.trial039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-25T08:51:34.534 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial039.front.sepia.ceph.com 2026-02-25T08:51:34.534 DEBUG:teuthology.orchestra.run.trial039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-25T08:51:34.600 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial091.front.sepia.ceph.com 2026-02-25T08:51:34.601 DEBUG:teuthology.orchestra.run.trial091:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-25T08:51:34.641 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial091.front.sepia.ceph.com 2026-02-25T08:51:34.641 DEBUG:teuthology.orchestra.run.trial091:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-25T08:51:34.708 INFO:tasks.ceph:Archiving mon data... 2026-02-25T08:51:34.709 DEBUG:teuthology.misc:Transferring archived files from trial039:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062/data/mon.a.tgz 2026-02-25T08:51:34.710 DEBUG:teuthology.orchestra.run.trial039:> mktemp 2026-02-25T08:51:34.727 INFO:teuthology.orchestra.run.trial039.stdout:/tmp/tmp.AueT5eNj9m 2026-02-25T08:51:34.728 DEBUG:teuthology.orchestra.run.trial039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.AueT5eNj9m 2026-02-25T08:51:34.839 DEBUG:teuthology.orchestra.run.trial039:> sudo chmod 0666 /tmp/tmp.AueT5eNj9m 2026-02-25T08:51:34.922 DEBUG:teuthology.orchestra.remote:trial039:/tmp/tmp.AueT5eNj9m is 398KB 2026-02-25T08:51:35.015 DEBUG:teuthology.orchestra.run.trial039:> rm -fr /tmp/tmp.AueT5eNj9m 2026-02-25T08:51:35.031 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-25T08:51:35.032 DEBUG:teuthology.orchestra.run.trial039:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T08:51:35.073 DEBUG:teuthology.orchestra.run.trial091:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T08:51:35.151 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T08:51:35.167 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T08:51:35.167 INFO:tasks.ceph:Archiving crash dumps... 2026-02-25T08:51:35.170 DEBUG:teuthology.misc:Transferring archived files from trial039:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062/remote/trial039/crash 2026-02-25T08:51:35.170 DEBUG:teuthology.orchestra.run.trial039:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T08:51:35.215 DEBUG:teuthology.misc:Transferring archived files from trial091:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062/remote/trial091/crash 2026-02-25T08:51:35.217 DEBUG:teuthology.orchestra.run.trial091:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T08:51:35.242 INFO:tasks.ceph:Compressing logs... 2026-02-25T08:51:35.242 DEBUG:teuthology.orchestra.run.trial039:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:51:35.260 DEBUG:teuthology.orchestra.run.trial091:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42692.log 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47824.log 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42692.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42692.log.gz 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42208.log 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47554.log/var/log/ceph/ceph-client.admin.47824.log: 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47824.log.gz 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42208.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42208.log.gz 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49013.log 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47554.log.gz 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48094.log 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49013.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49013.log.gz 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41330.log 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43449.log 2026-02-25T08:51:35.308 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48094.log: /var/log/ceph/ceph-client.admin.41330.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48094.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 49.6% 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41330.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47986.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43449.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42307.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41936.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47986.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41838.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42307.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41936.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.41936.log.gz --verbose 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: -- /var/log/ceph/ceph-client.admin.43827.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41838.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41838.log.gzgzip 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46636.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43827.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43827.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45609.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46636.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46636.log.gzgzip 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42530.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48202.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45609.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47392.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42530.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43071.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48202.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46204.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47392.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48689.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43071.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46204.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46204.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43719.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47608.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48689.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43719.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43719.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49663.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42434.log 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47608.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47608.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49663.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49663.log.gz 2026-02-25T08:51:35.309 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49284.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41473.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42434.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49284.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49284.log.gz -5 --verbose 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: -- /var/log/ceph/ceph-client.admin.44421.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41642.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41473.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44421.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44421.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47770.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46474.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41642.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48797.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47770.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46474.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46474.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40641.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40857.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48797.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40641.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40641.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47716.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48959.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40857.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47716.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.47716.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42385.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40334.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: 48.8% -- replaced with /var/log/ceph/ceph-client.admin.48959.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42385.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42385.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46798.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48635.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40334.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40334.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47446.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46798.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48635.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48635.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42584.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46150.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47446.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42584.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.42584.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41547.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41133.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46150.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41547.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41547.log.gz 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.39134.log 2026-02-25T08:51:35.310 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47122.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41133.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph.tmp-client.admin.39134.log: 67.5% -- replaced with /var/log/ceph/ceph.tmp-client.admin.39134.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48148.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43557.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47122.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48148.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48148.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41135.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45501.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43557.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41135.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41135.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49067.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46366.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45501.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46744.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49067.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46366.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46366.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46690.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46744.log: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.46744.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45339.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42132.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.46690.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45339.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45339.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47014.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43935.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42132.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47014.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.47014.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43017.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46906.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43935.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43017.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43017.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46960.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46906.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46906.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41132.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42638.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46960.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41132.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.41132.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48743.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48580.log 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42638.log.gz 2026-02-25T08:51:35.311 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49122.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.48743.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44205.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48580.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42746.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49122.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42034.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44205.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46258.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42746.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48040.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42034.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41134.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46258.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48040.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48040.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48418.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40287.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41134.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48418.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48418.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45555.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43233.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40287.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40568.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45555.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43233.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43233.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46312.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44745.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40568.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46312.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46312.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45447.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49895.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44745.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45447.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45447.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47338.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41741.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.49895.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47338.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46420.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41741.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48256.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42908.log 2026-02-25T08:51:35.312 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46420.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47932.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48256.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47662.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42908.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40792.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47932.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47662.log: gzip 49.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.47662.log.gz --verbose 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr: -- /var/log/ceph/ceph-client.admin.41669.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45285.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.40792.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43611.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41669.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45285.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45285.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44637.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43611.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43611.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47500.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44637.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44637.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40907.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47500.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47500.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47878.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40907.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49501.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48364.log 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47878.log.gz 2026-02-25T08:51:35.313 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49501.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49501.log.gz 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39504.log 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39410.log 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.39504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39557.log 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39504.log.gz 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr:/var/log/ceph/ceph-client.admin.39410.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39410.log.gz 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr:/var/log/ceph/ceph-client.admin.39557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39457.log 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39557.log.gz 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-25T08:51:35.315 INFO:teuthology.orchestra.run.trial091.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.39457.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39457.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46852.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42481.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48364.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46852.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46852.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48905.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.0.42481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44907.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.0.42481.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48905.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48905.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49392.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47176.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44907.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49392.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49392.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45069.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44961.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47176.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45069.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45069.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41546.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45393.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44961.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43395.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41546.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43341.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45393.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43395.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43395.log.gz 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43341.log: gzip 50.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.43341.log.gz --verbose 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: -- /var/log/ceph/ceph-client.admin.41985.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47230.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41985.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41985.log.gzgzip 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41331.log 2026-02-25T08:51:35.318 INFO:teuthology.orchestra.run.trial039.stderr: 87.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47230.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47230.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40201.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49338.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41331.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40201.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40201.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48851.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49446.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49338.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48851.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48851.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42854.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49176.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49446.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42854.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42854.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45879.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49176.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49176.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45879.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45879.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45123.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45663.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45123.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45123.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45015.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47068.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45663.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45015.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45015.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48526.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45987.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47068.log: /var/log/ceph/ceph-client.admin.48526.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47068.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48526.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43179.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44367.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.45987.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43179.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43179.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43503.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49825.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44367.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41446.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43503.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49825.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49825.log.gz 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46096.log 2026-02-25T08:51:35.319 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45231.log 2026-02-25T08:51:35.320 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41446.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46096.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46096.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44259.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42308.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45231.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44259.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.44259.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44151.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40767.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42308.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44151.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.44151.log.gz --verbose 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr: -- /var/log/ceph/ceph-client.admin.41005.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40767.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40767.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46528.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41005.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41005.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45177.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46528.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46528.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45825.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45177.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45177.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43125.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45825.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45825.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41743.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43125.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43125.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49230.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41743.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41743.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42800.log 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49230.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49230.log.gz 2026-02-25T08:51:35.321 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44853.log 2026-02-25T08:51:35.322 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42800.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42800.log.gz 2026-02-25T08:51:35.322 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43665.log 2026-02-25T08:51:35.322 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44853.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44853.log.gz 2026-02-25T08:51:35.322 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44583.log 2026-02-25T08:51:35.322 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43665.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.43665.log.gz 2026-02-25T08:51:35.322 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43287.log 2026-02-25T08:51:35.322 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44583.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44583.log.gz 2026-02-25T08:51:35.322 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42083.log 2026-02-25T08:51:35.322 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43287.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43287.log.gz -5 2026-02-25T08:51:35.322 INFO:teuthology.orchestra.run.trial039.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48310.log 2026-02-25T08:51:35.323 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42083.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42083.log.gz 2026-02-25T08:51:35.323 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46582.log 2026-02-25T08:51:35.323 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45771.log 2026-02-25T08:51:35.323 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48310.log.gz 2026-02-25T08:51:35.323 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46582.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46582.log.gz 2026-02-25T08:51:35.323 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48472.log 2026-02-25T08:51:35.323 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45771.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45771.log.gz 2026-02-25T08:51:35.323 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46041.log 2026-02-25T08:51:35.323 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.48472.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48472.log.gz 2026-02-25T08:51:35.323 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41332.log 2026-02-25T08:51:35.324 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.46041.log: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.46041.log.gz 2026-02-25T08:51:35.324 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44529.log 2026-02-25T08:51:35.324 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41332.log: gzip -5 49.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.41332.log.gz -- 2026-02-25T08:51:35.324 INFO:teuthology.orchestra.run.trial039.stderr: /var/log/ceph/ceph-client.admin.44475.log 2026-02-25T08:51:35.324 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44529.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44529.log.gz 2026-02-25T08:51:35.324 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49771.log 2026-02-25T08:51:35.324 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44475.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44475.log.gz 2026-02-25T08:51:35.324 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49609.log 2026-02-25T08:51:35.325 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44799.log 2026-02-25T08:51:35.325 INFO:teuthology.orchestra.run.trial039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49771.log.gz 2026-02-25T08:51:35.325 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49609.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49609.log.gz 2026-02-25T08:51:35.325 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44097.log 2026-02-25T08:51:35.325 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44799.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44799.log.gz 2026-02-25T08:51:35.325 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44313.log 2026-02-25T08:51:35.325 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44097.log: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.44097.log.gz 2026-02-25T08:51:35.325 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45933.log 2026-02-25T08:51:35.326 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44313.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44313.log.gz 2026-02-25T08:51:35.326 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44043.log 2026-02-25T08:51:35.326 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43773.log 2026-02-25T08:51:35.326 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45933.log.gz 2026-02-25T08:51:35.326 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44043.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44043.log.gz 2026-02-25T08:51:35.326 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-25T08:51:35.326 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43773.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43773.log.gz 2026-02-25T08:51:35.326 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43989.log 2026-02-25T08:51:35.326 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2026-02-25T08:51:35.327 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43989.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43989.log.gz 2026-02-25T08:51:35.327 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45717.log 2026-02-25T08:51:35.327 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44691.log 2026-02-25T08:51:35.327 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 2026-02-25T08:51:35.327 INFO:teuthology.orchestra.run.trial039.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-25T08:51:35.327 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.45717.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45717.log.gz 2026-02-25T08:51:35.327 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42962.log 2026-02-25T08:51:35.328 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.44691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49717.log 2026-02-25T08:51:35.328 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44691.log.gz 2026-02-25T08:51:35.328 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.42962.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42962.log.gz 2026-02-25T08:51:35.328 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47284.log 2026-02-25T08:51:35.328 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49717.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49717.log.gz 2026-02-25T08:51:35.328 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40542.log 2026-02-25T08:51:35.328 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.47284.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47284.log.gz 2026-02-25T08:51:35.329 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49555.log 2026-02-25T08:51:35.329 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.40542.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40542.log.gz 2026-02-25T08:51:35.329 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41887.log 2026-02-25T08:51:35.329 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49555.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49555.log.gz 2026-02-25T08:51:35.329 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41250.log 2026-02-25T08:51:35.329 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41887.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41887.log.gz 2026-02-25T08:51:35.329 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49932.log 2026-02-25T08:51:35.330 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.41250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43881.log 2026-02-25T08:51:35.330 INFO:teuthology.orchestra.run.trial039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41250.log.gz 2026-02-25T08:51:35.330 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.49932.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.49932.log.gz 2026-02-25T08:51:35.330 INFO:teuthology.orchestra.run.trial039.stderr:/var/log/ceph/ceph-client.admin.43881.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43881.log.gz 2026-02-25T08:51:35.340 INFO:teuthology.orchestra.run.trial091.stderr:/var/log/ceph/ceph-mgr.x.log: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-25T08:51:35.436 INFO:teuthology.orchestra.run.trial039.stderr: 89.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-25T08:51:35.740 INFO:teuthology.orchestra.run.trial091.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-25T08:51:36.465 INFO:teuthology.orchestra.run.trial091.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-25T08:51:36.467 INFO:teuthology.orchestra.run.trial091.stderr: 2026-02-25T08:51:36.467 INFO:teuthology.orchestra.run.trial091.stderr:real 0m1.166s 2026-02-25T08:51:36.467 INFO:teuthology.orchestra.run.trial091.stderr:user 0m1.528s 2026-02-25T08:51:36.467 INFO:teuthology.orchestra.run.trial091.stderr:sys 0m0.105s 2026-02-25T08:51:36.954 INFO:teuthology.orchestra.run.trial039.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-25T08:51:37.101 INFO:teuthology.orchestra.run.trial039.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-25T08:51:37.104 INFO:teuthology.orchestra.run.trial039.stderr: 2026-02-25T08:51:37.104 INFO:teuthology.orchestra.run.trial039.stderr:real 0m1.830s 2026-02-25T08:51:37.104 INFO:teuthology.orchestra.run.trial039.stderr:user 0m3.476s 2026-02-25T08:51:37.104 INFO:teuthology.orchestra.run.trial039.stderr:sys 0m0.260s 2026-02-25T08:51:37.104 INFO:tasks.ceph:Archiving logs... 2026-02-25T08:51:37.105 DEBUG:teuthology.misc:Transferring archived files from trial039:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062/remote/trial039/log 2026-02-25T08:51:37.106 DEBUG:teuthology.orchestra.run.trial039:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T08:51:37.621 DEBUG:teuthology.misc:Transferring archived files from trial091:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062/remote/trial091/log 2026-02-25T08:51:37.622 DEBUG:teuthology.orchestra.run.trial091:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T08:51:37.855 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial039 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-25T08:51:37.857 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:51:37.868 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T08:51:37.983 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:51:37.995 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T08:51:37.995 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-25T08:51:37.995 DEBUG:teuthology.orchestra.run.trial039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T08:51:37.997 DEBUG:teuthology.orchestra.run.trial091:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T08:51:38.023 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-25T08:51:38.034 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-25T08:51:38.035 DEBUG:teuthology.orchestra.run.trial039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:51:38.064 DEBUG:teuthology.orchestra.run.trial091:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:51:38.076 INFO:teuthology.orchestra.run.trial039.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:51:38.079 INFO:teuthology.orchestra.run.trial091.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:51:38.096 INFO:teuthology.orchestra.run.trial039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:51:38.096 INFO:teuthology.orchestra.run.trial039.stdout:=============================================================================== 2026-02-25T08:51:38.096 INFO:teuthology.orchestra.run.trial039.stdout:^+ kc1cloud.147.72.155.23.m> 3 8 377 204 -479us[ -550us] +/- 43ms 2026-02-25T08:51:38.097 INFO:teuthology.orchestra.run.trial039.stdout:^+ nu.binary.net 2 8 377 209 -783us[ -854us] +/- 53ms 2026-02-25T08:51:38.097 INFO:teuthology.orchestra.run.trial039.stdout:^* 108.61.215.221.vultruser> 2 8 377 199 -463us[ -534us] +/- 23ms 2026-02-25T08:51:38.097 INFO:teuthology.orchestra.run.trial039.stdout:^+ ntp3.radio-sunshine.org 2 7 377 71 +1600us[+1600us] +/- 68ms 2026-02-25T08:51:38.118 INFO:teuthology.orchestra.run.trial091.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:51:38.118 INFO:teuthology.orchestra.run.trial091.stdout:=============================================================================== 2026-02-25T08:51:38.118 INFO:teuthology.orchestra.run.trial091.stdout:^* ntp1.glypnod.com 2 8 377 203 +985us[ +931us] +/- 43ms 2026-02-25T08:51:38.118 INFO:teuthology.orchestra.run.trial091.stdout:^- 23.186.168.133 2 7 377 256 +1344us[+1292us] +/- 56ms 2026-02-25T08:51:38.118 INFO:teuthology.orchestra.run.trial091.stdout:^+ 172-233-189-68.ip.linode> 2 8 377 464 -2434us[-2473us] +/- 85ms 2026-02-25T08:51:38.118 INFO:teuthology.orchestra.run.trial091.stdout:^- 23.186.168.132 2 8 377 194 +1374us[+1374us] +/- 56ms 2026-02-25T08:51:38.119 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-25T08:51:38.134 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-25T08:51:38.135 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-25T08:51:38.156 DEBUG:teuthology.orchestra.run.trial039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T08:51:38.212 DEBUG:teuthology.orchestra.run.trial091:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T08:51:38.268 DEBUG:teuthology.orchestra.run.trial039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:51:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:51:38.296 DEBUG:teuthology.orchestra.run.trial091:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:51:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:51:38.323 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-25T08:51:38.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-25T08:51:38.358 INFO:teuthology.task.internal:Duration was 2754.064607 seconds 2026-02-25T08:51:38.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-25T08:51:38.375 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-25T08:51:38.376 DEBUG:teuthology.orchestra.run.trial039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T08:51:38.379 DEBUG:teuthology.orchestra.run.trial091:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T08:51:38.414 INFO:teuthology.orchestra.run.trial039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:51:38.417 INFO:teuthology.orchestra.run.trial091.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:51:38.833 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-25T08:51:38.833 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial039.front.sepia.ceph.com 2026-02-25T08:51:38.834 DEBUG:teuthology.orchestra.run.trial039:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T08:51:38.854 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial091.front.sepia.ceph.com 2026-02-25T08:51:38.855 DEBUG:teuthology.orchestra.run.trial091:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T08:51:38.893 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-25T08:51:38.893 DEBUG:teuthology.orchestra.run.trial039:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:51:38.896 DEBUG:teuthology.orchestra.run.trial091:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:51:38.998 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-25T08:51:38.998 DEBUG:teuthology.orchestra.run.trial039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:51:39.001 DEBUG:teuthology.orchestra.run.trial091:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:51:39.021 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:51:39.023 INFO:teuthology.orchestra.run.trial039.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:51:39.023 INFO:teuthology.orchestra.run.trial039.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:51:39.023 INFO:teuthology.orchestra.run.trial039.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T08:51:39.023 INFO:teuthology.orchestra.run.trial039.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T08:51:39.024 INFO:teuthology.orchestra.run.trial091.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:51:39.025 INFO:teuthology.orchestra.run.trial091.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:51:39.025 INFO:teuthology.orchestra.run.trial091.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T08:51:39.026 INFO:teuthology.orchestra.run.trial091.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:51:39.026 INFO:teuthology.orchestra.run.trial091.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T08:51:39.033 INFO:teuthology.orchestra.run.trial039.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T08:51:39.035 INFO:teuthology.orchestra.run.trial091.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T08:51:39.037 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-25T08:51:39.049 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-25T08:51:39.049 DEBUG:teuthology.orchestra.run.trial039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T08:51:39.098 DEBUG:teuthology.orchestra.run.trial091:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T08:51:39.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-25T08:51:39.143 DEBUG:teuthology.orchestra.run.trial039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:51:39.146 DEBUG:teuthology.orchestra.run.trial091:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:51:39.166 INFO:teuthology.orchestra.run.trial039.stdout:kernel.core_pattern = core 2026-02-25T08:51:39.189 INFO:teuthology.orchestra.run.trial091.stdout:kernel.core_pattern = core 2026-02-25T08:51:39.206 DEBUG:teuthology.orchestra.run.trial039:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:51:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:51:39.240 DEBUG:teuthology.orchestra.run.trial091:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:51:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:51:39.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-25T08:51:39.283 INFO:teuthology.task.internal:Transferring archived files... 2026-02-25T08:51:39.285 DEBUG:teuthology.misc:Transferring archived files from trial039:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062/remote/trial039 2026-02-25T08:51:39.285 DEBUG:teuthology.orchestra.run.trial039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T08:51:39.322 DEBUG:teuthology.misc:Transferring archived files from trial091:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69062/remote/trial091 2026-02-25T08:51:39.322 DEBUG:teuthology.orchestra.run.trial091:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T08:51:39.352 INFO:teuthology.task.internal:Removing archive directory... 2026-02-25T08:51:39.352 DEBUG:teuthology.orchestra.run.trial039:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T08:51:39.355 DEBUG:teuthology.orchestra.run.trial091:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T08:51:39.407 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-25T08:51:39.427 INFO:teuthology.task.internal:Not uploading archives. 2026-02-25T08:51:39.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-25T08:51:39.446 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-25T08:51:39.446 DEBUG:teuthology.orchestra.run.trial039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T08:51:39.449 DEBUG:teuthology.orchestra.run.trial091:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T08:51:39.463 INFO:teuthology.orchestra.run.trial039.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 08:51 /home/ubuntu/cephtest 2026-02-25T08:51:39.464 INFO:teuthology.orchestra.run.trial091.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 08:51 /home/ubuntu/cephtest 2026-02-25T08:51:39.465 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-25T08:51:39.478 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-25T08:51:39.494 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-25T08:51:39.498 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} thrashers/{default} thrashosds-health workloads/snaps-few-objects-balanced} duration: 2754.0646073818207 failure_reason: The WatchDog barked due to Daemon ceph.osd.2 has failed flavor: debug owner: scheduled_matan@soko04.front.sepia.ceph.com status: fail success: false 2026-02-25T08:51:39.499 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:51:39.606 INFO:teuthology.run:FAIL