2026-02-20T11:50:16.634 INFO:root:teuthology version: 1.2.4.dev11+ge6d9099ce 2026-02-20T11:50:16.649 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-20T11:50:16.730 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590 branch: wip-perezjos-crimson-only-19-02-2026-PR64439 description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} thrashers/{default} thrashosds-health workloads/snaps-few-objects} email: null first_in_suite: false flavor: debug job_id: '60590' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-perezjos-crimson-only-19-02-2026-PR64439 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_max_pg_log_entries: 2 osd_min_pg_log_entries: 1 osd_pg_log_trim_min: 0 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: seastore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd target transaction size: 5 seastore cachepin size pershard: 64M seastore cachepin type: 2Q seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Degraded data redundancy - is down - osds down - pg .*? is .*?degraded.*?, acting - pg .*? is stuck - pg degraded - PG_BACKFILL_FULL - Low space hindering backfill .*? backfill_toofull - OSD_ROOT_DOWN - pgs degraded - pgs undersized - is active.*backfill_toofull.* sha1: 0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: debug sha1: 0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 thrashosds: min_in: 4 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 445d5c9ba18a4040e70e25303230c36c77773acb owner: scheduled_jjperez@soko04.front.sepia.ceph.com priority: 101 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 1736 sha1: 0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 445d5c9ba18a4040e70e25303230c36c77773acb targets: trial153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOiFzemx1T/+ABLF5v0t5hoKbNWa4IQRyZtGZE4xgH2BHebCKfBoD+PQGjP2XPt+sdgvKtz4qcRVhJz4PcsaIcc= trial183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCZAv4U83dWpuIrSXfTRzNldrJbHj71xsfPntOMn5d6Jq/rvPqvFoMpv52jgAX28bKUEVLeFT/9KCWOjkrAktQU= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_trim_stale_osdmaps: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: reimage-ssh-down teuthology_sha1: e6d9099ce03dc21193fda653159ec82465c3a94a timestamp: 2026-02-20_10:21:31 tube: trial user: jjperez verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699 2026-02-20T11:50:16.730 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa; will attempt to use it 2026-02-20T11:50:16.731 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks 2026-02-20T11:50:16.731 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-20T11:50:16.732 INFO:teuthology.task.internal:Checking packages... 2026-02-20T11:50:16.732 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4' 2026-02-20T11:50:16.733 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-20T11:50:16.733 INFO:teuthology.packaging:ref: None 2026-02-20T11:50:16.733 INFO:teuthology.packaging:tag: None 2026-02-20T11:50:16.733 INFO:teuthology.packaging:branch: wip-perezjos-crimson-only-19-02-2026-PR64439 2026-02-20T11:50:16.733 INFO:teuthology.packaging:sha1: 0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:50:16.733 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-perezjos-crimson-only-19-02-2026-PR64439 2026-02-20T11:50:16.851 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5357.g0c3148ca 2026-02-20T11:50:16.852 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-20T11:50:16.863 INFO:teuthology.task.internal:no buildpackages task found 2026-02-20T11:50:16.864 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-20T11:50:16.876 INFO:teuthology.task.internal:Saving configuration 2026-02-20T11:50:16.884 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-20T11:50:16.895 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-20T11:50:16.952 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial153.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-20 11:46:47.322758', 'locked_by': 'scheduled_jjperez@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOiFzemx1T/+ABLF5v0t5hoKbNWa4IQRyZtGZE4xgH2BHebCKfBoD+PQGjP2XPt+sdgvKtz4qcRVhJz4PcsaIcc='} 2026-02-20T11:50:17.007 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial183.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-20 11:46:47.322075', 'locked_by': 'scheduled_jjperez@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCZAv4U83dWpuIrSXfTRzNldrJbHj71xsfPntOMn5d6Jq/rvPqvFoMpv52jgAX28bKUEVLeFT/9KCWOjkrAktQU='} 2026-02-20T11:50:17.007 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-20T11:50:17.019 INFO:teuthology.task.internal:roles: ubuntu@trial153.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2026-02-20T11:50:17.019 INFO:teuthology.task.internal:roles: ubuntu@trial183.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2026-02-20T11:50:17.019 INFO:teuthology.run_tasks:Running task console_log... 2026-02-20T11:50:17.147 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7e9651c4f600>, signals=[15]) 2026-02-20T11:50:17.147 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-20T11:50:17.158 INFO:teuthology.task.internal:Opening connections... 2026-02-20T11:50:17.158 DEBUG:teuthology.task.internal:connecting to ubuntu@trial153.front.sepia.ceph.com 2026-02-20T11:50:17.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T11:50:17.219 DEBUG:teuthology.task.internal:connecting to ubuntu@trial183.front.sepia.ceph.com 2026-02-20T11:50:17.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T11:50:17.280 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-20T11:50:17.291 DEBUG:teuthology.orchestra.run.trial153:> uname -m 2026-02-20T11:50:17.306 INFO:teuthology.orchestra.run.trial153.stdout:x86_64 2026-02-20T11:50:17.307 DEBUG:teuthology.orchestra.run.trial153:> cat /etc/os-release 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:NAME="CentOS Stream" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:VERSION="9" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:ID="centos" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:ID_LIKE="rhel fedora" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:VERSION_ID="9" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:PLATFORM_ID="platform:el9" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:ANSI_COLOR="0;31" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:LOGO="fedora-logo-icon" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:HOME_URL="https://centos.org/" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-20T11:50:17.363 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-20T11:50:17.364 INFO:teuthology.lock.ops:Updating trial153.front.sepia.ceph.com on lock server 2026-02-20T11:50:17.434 DEBUG:teuthology.orchestra.run.trial183:> uname -m 2026-02-20T11:50:17.450 INFO:teuthology.orchestra.run.trial183.stdout:x86_64 2026-02-20T11:50:17.450 DEBUG:teuthology.orchestra.run.trial183:> cat /etc/os-release 2026-02-20T11:50:17.505 INFO:teuthology.orchestra.run.trial183.stdout:NAME="CentOS Stream" 2026-02-20T11:50:17.505 INFO:teuthology.orchestra.run.trial183.stdout:VERSION="9" 2026-02-20T11:50:17.505 INFO:teuthology.orchestra.run.trial183.stdout:ID="centos" 2026-02-20T11:50:17.505 INFO:teuthology.orchestra.run.trial183.stdout:ID_LIKE="rhel fedora" 2026-02-20T11:50:17.505 INFO:teuthology.orchestra.run.trial183.stdout:VERSION_ID="9" 2026-02-20T11:50:17.505 INFO:teuthology.orchestra.run.trial183.stdout:PLATFORM_ID="platform:el9" 2026-02-20T11:50:17.505 INFO:teuthology.orchestra.run.trial183.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-20T11:50:17.505 INFO:teuthology.orchestra.run.trial183.stdout:ANSI_COLOR="0;31" 2026-02-20T11:50:17.505 INFO:teuthology.orchestra.run.trial183.stdout:LOGO="fedora-logo-icon" 2026-02-20T11:50:17.505 INFO:teuthology.orchestra.run.trial183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-20T11:50:17.506 INFO:teuthology.orchestra.run.trial183.stdout:HOME_URL="https://centos.org/" 2026-02-20T11:50:17.506 INFO:teuthology.orchestra.run.trial183.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-20T11:50:17.506 INFO:teuthology.orchestra.run.trial183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-20T11:50:17.506 INFO:teuthology.orchestra.run.trial183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-20T11:50:17.506 INFO:teuthology.lock.ops:Updating trial183.front.sepia.ceph.com on lock server 2026-02-20T11:50:17.582 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-20T11:50:17.593 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-20T11:50:17.604 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-20T11:50:17.604 DEBUG:teuthology.orchestra.run.trial153:> test '!' -e /home/ubuntu/cephtest 2026-02-20T11:50:17.606 DEBUG:teuthology.orchestra.run.trial183:> test '!' -e /home/ubuntu/cephtest 2026-02-20T11:50:17.621 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-20T11:50:17.632 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-20T11:50:17.632 DEBUG:teuthology.orchestra.run.trial153:> test -z $(ls -A /var/lib/ceph) 2026-02-20T11:50:17.661 DEBUG:teuthology.orchestra.run.trial183:> test -z $(ls -A /var/lib/ceph) 2026-02-20T11:50:17.678 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-20T11:50:17.836 INFO:teuthology.run_tasks:Running task kernel... 2026-02-20T11:50:17.856 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-20T11:50:17.856 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-20T11:50:17.856 DEBUG:teuthology.orchestra.run.trial153:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-20T11:50:17.856 DEBUG:teuthology.orchestra.run.trial183:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-20T11:50:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:50:17.871 DEBUG:teuthology.orchestra.run.trial153:> uname -r 2026-02-20T11:50:17.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:50:17.871 DEBUG:teuthology.orchestra.run.trial183:> uname -r 2026-02-20T11:50:17.926 INFO:teuthology.orchestra.run.trial153.stdout:5.14.0-681.el9.x86_64 2026-02-20T11:50:17.926 INFO:teuthology.task.kernel:Running kernel on trial153: 5.14.0-681.el9.x86_64 2026-02-20T11:50:17.926 DEBUG:teuthology.orchestra.run.trial153:> sudo yum install -y kernel 2026-02-20T11:50:17.927 INFO:teuthology.orchestra.run.trial183.stdout:5.14.0-681.el9.x86_64 2026-02-20T11:50:17.927 INFO:teuthology.task.kernel:Running kernel on trial183: 5.14.0-681.el9.x86_64 2026-02-20T11:50:17.927 DEBUG:teuthology.orchestra.run.trial183:> sudo yum install -y kernel 2026-02-20T11:50:18.997 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.9 MB 00:00 2026-02-20T11:50:19.103 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-20T11:50:22.822 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - AppStream 8.4 MB/s | 27 MB 00:03 2026-02-20T11:50:26.059 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 7.8 MB 00:00 2026-02-20T11:50:27.194 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - AppStream 3.5 MB/s | 27 MB 00:07 2026-02-20T11:50:27.491 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 20 kB 00:00 2026-02-20T11:50:28.275 INFO:teuthology.orchestra.run.trial153.stdout:Extra Packages for Enterprise Linux 28 MB/s | 20 MB 00:00 2026-02-20T11:50:31.478 INFO:teuthology.orchestra.run.trial153.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-20T11:50:32.489 INFO:teuthology.orchestra.run.trial153.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-20T11:50:32.489 INFO:teuthology.orchestra.run.trial153.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-20T11:50:32.508 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:50:32.509 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:50:32.510 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:50:32.533 DEBUG:teuthology.orchestra.run.trial153:> echo no | sudo yum reinstall kernel || true 2026-02-20T11:50:32.830 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:01 ago on Fri 20 Feb 2026 11:50:31 AM UTC. 2026-02-20T11:50:32.909 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout: Package Architecture Version Repository Size 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout:Reinstalling: 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 803 k 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 0 2026-02-20T11:50:32.910 INFO:teuthology.orchestra.run.trial153.stderr:Operation aborted. 2026-02-20T11:50:32.929 INFO:teuthology.orchestra.run.trial153.stdout:Is this ok [y/N]: 2026-02-20T11:50:32.929 DEBUG:teuthology.orchestra.run.trial153:> sudo yum reinstall -y kernel || true 2026-02-20T11:50:33.119 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 7.8 MB 00:03 2026-02-20T11:50:33.228 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:50:31 AM UTC. 2026-02-20T11:50:33.309 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:50:33.309 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:50:33.309 INFO:teuthology.orchestra.run.trial153.stdout: Package Architecture Version Repository Size 2026-02-20T11:50:33.309 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:50:33.310 INFO:teuthology.orchestra.run.trial153.stdout:Reinstalling: 2026-02-20T11:50:33.310 INFO:teuthology.orchestra.run.trial153.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-20T11:50:33.310 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:50:33.310 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:50:33.310 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:50:33.310 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:50:33.310 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 803 k 2026-02-20T11:50:33.310 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 0 2026-02-20T11:50:33.310 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:50:33.527 INFO:teuthology.orchestra.run.trial153.stdout:kernel-5.14.0-681.el9.x86_64.rpm 5.7 MB/s | 803 kB 00:00 2026-02-20T11:50:33.528 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:50:33.528 INFO:teuthology.orchestra.run.trial153.stdout:Total 3.6 MB/s | 803 kB 00:00 2026-02-20T11:50:33.537 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:50:33.570 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:50:33.571 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:50:33.575 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:50:33.575 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:50:33.599 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:50:33.603 INFO:teuthology.orchestra.run.trial153.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-20T11:50:33.603 INFO:teuthology.orchestra.run.trial153.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:50:33.720 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:50:33.720 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-20T11:50:33.771 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:50:33.771 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:50:33.771 INFO:teuthology.orchestra.run.trial153.stdout:Reinstalled: 2026-02-20T11:50:33.771 INFO:teuthology.orchestra.run.trial153.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-20T11:50:33.771 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:50:33.771 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:50:33.799 DEBUG:teuthology.orchestra.run.trial153:> rpm -q kernel | sort -rV | head -n 1 2026-02-20T11:50:33.823 INFO:teuthology.orchestra.run.trial153.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-20T11:50:33.824 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-20T11:50:33.824 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-20T11:50:33.825 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-20T11:50:33.825 DEBUG:teuthology.orchestra.run.trial153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-20T11:50:33.888 INFO:teuthology.orchestra.run.trial153.stdout:ttyS1 2026-02-20T11:50:33.910 DEBUG:teuthology.parallel:result is None 2026-02-20T11:50:34.024 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - Extras packages 106 kB/s | 20 kB 00:00 2026-02-20T11:50:34.754 INFO:teuthology.orchestra.run.trial183.stdout:Extra Packages for Enterprise Linux 30 MB/s | 20 MB 00:00 2026-02-20T11:50:37.893 INFO:teuthology.orchestra.run.trial183.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-20T11:50:38.919 INFO:teuthology.orchestra.run.trial183.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-20T11:50:38.919 INFO:teuthology.orchestra.run.trial183.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-20T11:50:38.938 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:50:38.939 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:50:38.939 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:50:38.962 DEBUG:teuthology.orchestra.run.trial183:> echo no | sudo yum reinstall kernel || true 2026-02-20T11:50:39.259 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:50:37 AM UTC. 2026-02-20T11:50:39.338 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:50:39.340 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:50:39.340 INFO:teuthology.orchestra.run.trial183.stdout: Package Architecture Version Repository Size 2026-02-20T11:50:39.340 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:50:39.340 INFO:teuthology.orchestra.run.trial183.stdout:Reinstalling: 2026-02-20T11:50:39.340 INFO:teuthology.orchestra.run.trial183.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-20T11:50:39.340 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:50:39.340 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:50:39.340 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:50:39.340 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:50:39.340 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 803 k 2026-02-20T11:50:39.341 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 0 2026-02-20T11:50:39.341 INFO:teuthology.orchestra.run.trial183.stderr:Operation aborted. 2026-02-20T11:50:39.360 INFO:teuthology.orchestra.run.trial183.stdout:Is this ok [y/N]: 2026-02-20T11:50:39.361 DEBUG:teuthology.orchestra.run.trial183:> sudo yum reinstall -y kernel || true 2026-02-20T11:50:39.660 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:50:37 AM UTC. 2026-02-20T11:50:39.740 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:50:39.740 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:50:39.740 INFO:teuthology.orchestra.run.trial183.stdout: Package Architecture Version Repository Size 2026-02-20T11:50:39.740 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:50:39.740 INFO:teuthology.orchestra.run.trial183.stdout:Reinstalling: 2026-02-20T11:50:39.740 INFO:teuthology.orchestra.run.trial183.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-20T11:50:39.740 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:50:39.741 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:50:39.741 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:50:39.741 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:50:39.741 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 803 k 2026-02-20T11:50:39.741 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 0 2026-02-20T11:50:39.741 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:50:40.599 INFO:teuthology.orchestra.run.trial183.stdout:kernel-5.14.0-681.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2026-02-20T11:50:40.599 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:50:40.599 INFO:teuthology.orchestra.run.trial183.stdout:Total 935 kB/s | 803 kB 00:00 2026-02-20T11:50:40.609 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:50:40.641 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:50:40.642 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:50:40.646 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:50:40.646 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:50:40.670 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:50:40.674 INFO:teuthology.orchestra.run.trial183.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-20T11:50:40.674 INFO:teuthology.orchestra.run.trial183.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:50:40.785 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:50:40.785 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-20T11:50:40.843 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-20T11:50:40.843 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:50:40.843 INFO:teuthology.orchestra.run.trial183.stdout:Reinstalled: 2026-02-20T11:50:40.843 INFO:teuthology.orchestra.run.trial183.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-20T11:50:40.843 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:50:40.843 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:50:40.869 DEBUG:teuthology.orchestra.run.trial183:> rpm -q kernel | sort -rV | head -n 1 2026-02-20T11:50:40.895 INFO:teuthology.orchestra.run.trial183.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-20T11:50:40.895 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-20T11:50:40.895 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-20T11:50:40.896 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2026-02-20T11:50:40.896 DEBUG:teuthology.orchestra.run.trial183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-20T11:50:40.960 INFO:teuthology.orchestra.run.trial183.stdout:ttyS1 2026-02-20T11:50:40.985 DEBUG:teuthology.parallel:result is None 2026-02-20T11:50:40.985 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-20T11:50:40.996 INFO:teuthology.task.internal:Creating test directory... 2026-02-20T11:50:40.996 DEBUG:teuthology.orchestra.run.trial153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-20T11:50:40.998 DEBUG:teuthology.orchestra.run.trial183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-20T11:50:41.042 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-20T11:50:41.052 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-20T11:50:41.065 INFO:teuthology.task.internal:Creating archive directory... 2026-02-20T11:50:41.066 DEBUG:teuthology.orchestra.run.trial153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-20T11:50:41.067 DEBUG:teuthology.orchestra.run.trial183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-20T11:50:41.103 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-20T11:50:41.113 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-20T11:50:41.113 DEBUG:teuthology.orchestra.run.trial153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-20T11:50:41.124 DEBUG:teuthology.orchestra.run.trial183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-20T11:50:41.147 INFO:teuthology.orchestra.run.trial153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T11:50:41.156 INFO:teuthology.orchestra.run.trial153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T11:50:41.163 INFO:teuthology.orchestra.run.trial183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T11:50:41.172 INFO:teuthology.orchestra.run.trial183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T11:50:41.174 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-20T11:50:41.186 INFO:teuthology.task.internal:Configuring sudo... 2026-02-20T11:50:41.186 DEBUG:teuthology.orchestra.run.trial153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-20T11:50:41.199 DEBUG:teuthology.orchestra.run.trial183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-20T11:50:41.238 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-20T11:50:41.251 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-20T11:50:41.251 DEBUG:teuthology.orchestra.run.trial153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-20T11:50:41.264 DEBUG:teuthology.orchestra.run.trial183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-20T11:50:41.294 DEBUG:teuthology.orchestra.run.trial153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T11:50:41.435 DEBUG:teuthology.orchestra.run.trial153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T11:50:41.458 DEBUG:teuthology.orchestra.run.trial153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T11:50:41.603 DEBUG:teuthology.orchestra.run.trial153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T11:50:41.627 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:50:41.627 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-20T11:50:41.690 DEBUG:teuthology.orchestra.run.trial183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T11:50:41.791 DEBUG:teuthology.orchestra.run.trial183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T11:50:41.817 DEBUG:teuthology.orchestra.run.trial183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T11:50:42.023 DEBUG:teuthology.orchestra.run.trial183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T11:50:42.049 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:50:42.049 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-20T11:50:42.112 DEBUG:teuthology.orchestra.run.trial153:> sudo service rsyslog restart 2026-02-20T11:50:42.114 DEBUG:teuthology.orchestra.run.trial183:> sudo service rsyslog restart 2026-02-20T11:50:42.139 INFO:teuthology.orchestra.run.trial153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-20T11:50:42.179 INFO:teuthology.orchestra.run.trial183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-20T11:50:42.585 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-20T11:50:42.597 INFO:teuthology.task.internal:Starting timer... 2026-02-20T11:50:42.598 INFO:teuthology.run_tasks:Running task pcp... 2026-02-20T11:50:42.620 INFO:teuthology.run_tasks:Running task selinux... 2026-02-20T11:50:42.634 DEBUG:teuthology.orchestra.run.trial153:> sudo service auditd rotate 2026-02-20T11:50:42.662 INFO:teuthology.orchestra.run.trial153.stdout:Rotating logs: 2026-02-20T11:50:42.664 DEBUG:teuthology.orchestra.run.trial183:> sudo service auditd rotate 2026-02-20T11:50:42.694 INFO:teuthology.orchestra.run.trial183.stdout:Rotating logs: 2026-02-20T11:50:42.696 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-20T11:50:42.696 DEBUG:teuthology.orchestra.run.trial153:> /usr/sbin/getenforce 2026-02-20T11:50:42.718 INFO:teuthology.orchestra.run.trial153.stdout:Permissive 2026-02-20T11:50:42.719 DEBUG:teuthology.orchestra.run.trial183:> /usr/sbin/getenforce 2026-02-20T11:50:42.754 INFO:teuthology.orchestra.run.trial183.stdout:Permissive 2026-02-20T11:50:42.754 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial153.front.sepia.ceph.com': 'permissive', 'ubuntu@trial183.front.sepia.ceph.com': 'permissive'} 2026-02-20T11:50:42.754 DEBUG:teuthology.orchestra.run.trial153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-20T11:50:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:50:42.783 DEBUG:teuthology.orchestra.run.trial183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-20T11:50:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T11:50:42.819 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-20T11:50:42.820 DEBUG:teuthology.orchestra.run.trial153:> sudo /usr/sbin/setenforce permissive 2026-02-20T11:50:42.848 DEBUG:teuthology.orchestra.run.trial183:> sudo /usr/sbin/setenforce permissive 2026-02-20T11:50:42.885 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-20T11:50:42.902 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-20T11:50:42.906 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-20T11:50:43.112 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-20T11:50:43.118 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-20T11:50:43.119 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial153.front.sepia.ceph.com,trial183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-20T11:53:31.330 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial153.front.sepia.ceph.com'), Remote(name='ubuntu@trial183.front.sepia.ceph.com')] 2026-02-20T11:53:31.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial153.front.sepia.ceph.com' 2026-02-20T11:53:31.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T11:53:31.491 DEBUG:teuthology.orchestra.run.trial153:> true 2026-02-20T11:53:31.566 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial153.front.sepia.ceph.com' 2026-02-20T11:53:31.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-20T11:53:31.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T11:53:31.657 DEBUG:teuthology.orchestra.run.trial183:> true 2026-02-20T11:53:31.733 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-20T11:53:31.733 INFO:teuthology.run_tasks:Running task clock... 2026-02-20T11:53:31.751 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-20T11:53:31.752 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-20T11:53:31.752 DEBUG:teuthology.orchestra.run.trial153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T11:53:31.754 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-20T11:53:31.754 DEBUG:teuthology.orchestra.run.trial183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T11:53:31.781 INFO:teuthology.orchestra.run.trial153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-20T11:53:31.793 INFO:teuthology.orchestra.run.trial153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-20T11:53:31.800 INFO:teuthology.orchestra.run.trial183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-20T11:53:31.813 INFO:teuthology.orchestra.run.trial183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-20T11:53:31.815 INFO:teuthology.orchestra.run.trial153.stderr:sudo: ntpd: command not found 2026-02-20T11:53:31.824 INFO:teuthology.orchestra.run.trial153.stdout:506 Cannot talk to daemon 2026-02-20T11:53:31.834 INFO:teuthology.orchestra.run.trial183.stderr:sudo: ntpd: command not found 2026-02-20T11:53:31.835 INFO:teuthology.orchestra.run.trial153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-20T11:53:31.844 INFO:teuthology.orchestra.run.trial183.stdout:506 Cannot talk to daemon 2026-02-20T11:53:31.847 INFO:teuthology.orchestra.run.trial153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-20T11:53:31.855 INFO:teuthology.orchestra.run.trial183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-20T11:53:31.868 INFO:teuthology.orchestra.run.trial183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-20T11:53:31.894 INFO:teuthology.orchestra.run.trial153.stderr:bash: line 1: ntpq: command not found 2026-02-20T11:53:31.895 INFO:teuthology.orchestra.run.trial153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-20T11:53:31.895 INFO:teuthology.orchestra.run.trial153.stdout:=============================================================================== 2026-02-20T11:53:31.924 INFO:teuthology.orchestra.run.trial183.stderr:bash: line 1: ntpq: command not found 2026-02-20T11:53:31.925 INFO:teuthology.orchestra.run.trial183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-20T11:53:31.926 INFO:teuthology.orchestra.run.trial183.stdout:=============================================================================== 2026-02-20T11:53:31.927 INFO:teuthology.run_tasks:Running task install... 2026-02-20T11:53:31.943 DEBUG:teuthology.task.install:project ceph 2026-02-20T11:53:31.943 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}} 2026-02-20T11:53:31.943 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'} 2026-02-20T11:53:31.943 INFO:teuthology.task.install:Using flavor: debug 2026-02-20T11:53:31.952 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-20T11:53:31.953 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-20T11:53:31.953 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'tag': None, 'wait_for_package': False} 2026-02-20T11:53:31.954 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:53:31.957 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'tag': None, 'wait_for_package': False} 2026-02-20T11:53:31.958 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:53:32.146 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/ 2026-02-20T11:53:32.146 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5357.g0c3148ca 2026-02-20T11:53:32.156 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/ 2026-02-20T11:53:32.156 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5357.g0c3148ca 2026-02-20T11:53:32.406 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-20T11:53:32.406 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:53:32.406 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-20T11:53:32.432 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-20T11:53:32.432 DEBUG:teuthology.orchestra.run.trial153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-20T11:53:32.447 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-20T11:53:32.447 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:53:32.447 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-20T11:53:32.474 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-20T11:53:32.474 DEBUG:teuthology.orchestra.run.trial183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-20T11:53:32.496 DEBUG:teuthology.orchestra.run.trial153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-20T11:53:32.539 DEBUG:teuthology.orchestra.run.trial183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-20T11:53:32.570 DEBUG:teuthology.orchestra.run.trial153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-20T11:53:32.612 DEBUG:teuthology.orchestra.run.trial183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-20T11:53:32.633 INFO:teuthology.orchestra.run.trial153.stdout:check_obsoletes = 1 2026-02-20T11:53:32.634 DEBUG:teuthology.orchestra.run.trial153:> sudo yum clean all 2026-02-20T11:53:32.676 INFO:teuthology.orchestra.run.trial183.stdout:check_obsoletes = 1 2026-02-20T11:53:32.677 DEBUG:teuthology.orchestra.run.trial183:> sudo yum clean all 2026-02-20T11:53:32.793 INFO:teuthology.orchestra.run.trial153.stdout:41 files removed 2026-02-20T11:53:32.808 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-radosgw 2026-02-20T11:53:32.838 INFO:teuthology.orchestra.run.trial183.stdout:41 files removed 2026-02-20T11:53:32.853 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-radosgw 2026-02-20T11:53:33.552 INFO:teuthology.orchestra.run.trial153.stdout:ceph packages for x86_64 156 kB/s | 94 kB 00:00 2026-02-20T11:53:33.603 INFO:teuthology.orchestra.run.trial183.stdout:ceph packages for x86_64 154 kB/s | 94 kB 00:00 2026-02-20T11:53:34.185 INFO:teuthology.orchestra.run.trial153.stdout:ceph noarch packages 40 kB/s | 24 kB 00:00 2026-02-20T11:53:34.230 INFO:teuthology.orchestra.run.trial183.stdout:ceph noarch packages 41 kB/s | 24 kB 00:00 2026-02-20T11:53:34.663 INFO:teuthology.orchestra.run.trial153.stdout:ceph source packages 4.5 kB/s | 2.1 kB 00:00 2026-02-20T11:53:34.696 INFO:teuthology.orchestra.run.trial183.stdout:ceph source packages 4.7 kB/s | 2.1 kB 00:00 2026-02-20T11:53:35.708 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - BaseOS 9.0 MB/s | 8.9 MB 00:00 2026-02-20T11:53:37.665 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.9 MB 00:02 2026-02-20T11:53:38.785 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-20T11:53:42.620 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 7.8 MB 00:01 2026-02-20T11:53:43.253 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 27 MB 00:05 2026-02-20T11:53:43.481 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - Extras packages 158 kB/s | 20 kB 00:00 2026-02-20T11:53:45.161 INFO:teuthology.orchestra.run.trial183.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2026-02-20T11:53:46.667 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 7.8 MB 00:01 2026-02-20T11:53:48.475 INFO:teuthology.orchestra.run.trial183.stdout:lab-extras 514 kB/s | 50 kB 00:00 2026-02-20T11:53:48.684 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 20 kB 00:01 2026-02-20T11:53:49.084 INFO:teuthology.orchestra.run.trial153.stdout:Extra Packages for Enterprise Linux 60 MB/s | 20 MB 00:00 2026-02-20T11:53:49.660 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:53:49.661 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:53:49.661 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repository Size 2026-02-20T11:53:49.661 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:53:49.661 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:53:49.661 INFO:teuthology.orchestra.run.trial183.stdout: ceph-radosgw x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 11 M 2026-02-20T11:53:49.661 INFO:teuthology.orchestra.run.trial183.stdout:Upgrading: 2026-02-20T11:53:49.661 INFO:teuthology.orchestra.run.trial183.stdout: librados2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 3.3 M 2026-02-20T11:53:49.661 INFO:teuthology.orchestra.run.trial183.stdout: librbd1 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.7 M 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: ceph-base x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 5.1 M 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: ceph-common x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 21 M 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: ceph-selinux x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 25 k 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 860 k 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-20T11:53:49.662 INFO:teuthology.orchestra.run.trial183.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: libradosstriper1 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 263 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: librgw2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 5.6 M 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-argparse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 45 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-common x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 185 k 2026-02-20T11:53:49.663 INFO:teuthology.orchestra.run.trial183.stdout: python3-cephfs x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 204 k 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: python3-rados x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 324 k 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: python3-rbd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 305 k 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: python3-rgw x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 100 k 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout:Installing weak dependencies: 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-daemon x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 37 k 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 26 k 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-20T11:53:49.664 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:53:49.665 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:53:49.665 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:53:49.665 INFO:teuthology.orchestra.run.trial183.stdout:Install 35 Packages 2026-02-20T11:53:49.665 INFO:teuthology.orchestra.run.trial183.stdout:Upgrade 2 Packages 2026-02-20T11:53:49.665 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:53:49.666 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 61 M 2026-02-20T11:53:49.666 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:53:51.988 INFO:teuthology.orchestra.run.trial183.stdout:(1/37): ceph-base-20.3.0-5357.g0c3148ca.el9.x86 2.4 MB/s | 5.1 MB 00:02 2026-02-20T11:53:52.051 INFO:teuthology.orchestra.run.trial183.stdout:(2/37): ceph-selinux-20.3.0-5357.g0c3148ca.el9. 400 kB/s | 25 kB 00:00 2026-02-20T11:53:52.115 INFO:teuthology.orchestra.run.trial183.stdout:(3/37): libcephfs-daemon-20.3.0-5357.g0c3148ca. 574 kB/s | 37 kB 00:00 2026-02-20T11:53:52.180 INFO:teuthology.orchestra.run.trial183.stdout:(4/37): libcephfs-proxy2-20.3.0-5357.g0c3148ca. 396 kB/s | 26 kB 00:00 2026-02-20T11:53:52.282 INFO:teuthology.orchestra.run.trial153.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-20T11:53:52.315 INFO:teuthology.orchestra.run.trial183.stdout:(5/37): ceph-radosgw-20.3.0-5357.g0c3148ca.el9. 4.6 MB/s | 11 MB 00:02 2026-02-20T11:53:52.383 INFO:teuthology.orchestra.run.trial183.stdout:(6/37): libradosstriper1-20.3.0-5357.g0c3148ca. 3.8 MB/s | 263 kB 00:00 2026-02-20T11:53:52.444 INFO:teuthology.orchestra.run.trial183.stdout:(7/37): libcephfs2-20.3.0-5357.g0c3148ca.el9.x8 3.2 MB/s | 860 kB 00:00 2026-02-20T11:53:52.510 INFO:teuthology.orchestra.run.trial183.stdout:(8/37): python3-ceph-argparse-20.3.0-5357.g0c31 681 kB/s | 45 kB 00:00 2026-02-20T11:53:52.580 INFO:teuthology.orchestra.run.trial183.stdout:(9/37): python3-ceph-common-20.3.0-5357.g0c3148 2.6 MB/s | 185 kB 00:00 2026-02-20T11:53:52.653 INFO:teuthology.orchestra.run.trial183.stdout:(10/37): python3-cephfs-20.3.0-5357.g0c3148ca.e 2.7 MB/s | 204 kB 00:00 2026-02-20T11:53:52.793 INFO:teuthology.orchestra.run.trial183.stdout:(11/37): python3-rados-20.3.0-5357.g0c3148ca.el 2.3 MB/s | 324 kB 00:00 2026-02-20T11:53:52.906 INFO:teuthology.orchestra.run.trial183.stdout:(12/37): python3-rbd-20.3.0-5357.g0c3148ca.el9. 2.6 MB/s | 305 kB 00:00 2026-02-20T11:53:52.960 INFO:teuthology.orchestra.run.trial183.stdout:(13/37): python3-rgw-20.3.0-5357.g0c3148ca.el9. 1.8 MB/s | 100 kB 00:00 2026-02-20T11:53:53.117 INFO:teuthology.orchestra.run.trial183.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 211 kB/s | 33 kB 00:00 2026-02-20T11:53:53.271 INFO:teuthology.orchestra.run.trial183.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 7.5 MB/s | 1.2 MB 00:00 2026-02-20T11:53:53.400 INFO:teuthology.orchestra.run.trial183.stdout:(16/37): librgw2-20.3.0-5357.g0c3148ca.el9.x86_ 5.5 MB/s | 5.6 MB 00:01 2026-02-20T11:53:53.430 INFO:teuthology.orchestra.run.trial183.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 584 kB/s | 93 kB 00:00 2026-02-20T11:53:53.438 INFO:teuthology.orchestra.run.trial183.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 2.7 MB/s | 104 kB 00:00 2026-02-20T11:53:53.454 INFO:teuthology.orchestra.run.trial183.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 9.6 MB/s | 160 kB 00:00 2026-02-20T11:53:53.463 INFO:teuthology.orchestra.run.trial183.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 5.0 MB/s | 45 kB 00:00 2026-02-20T11:53:53.471 INFO:teuthology.orchestra.run.trial183.stdout:(21/37): libnbd-1.20.3-4.el9.x86_64.rpm 3.9 MB/s | 164 kB 00:00 2026-02-20T11:53:53.484 INFO:teuthology.orchestra.run.trial183.stdout:(22/37): librdkafka-1.6.1-102.el9.x86_64.rpm 32 MB/s | 662 kB 00:00 2026-02-20T11:53:53.486 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:53:53.488 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:53:53.488 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repository Size 2026-02-20T11:53:53.488 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:53:53.488 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:53:53.488 INFO:teuthology.orchestra.run.trial153.stdout: ceph-radosgw x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 11 M 2026-02-20T11:53:53.488 INFO:teuthology.orchestra.run.trial153.stdout:Upgrading: 2026-02-20T11:53:53.488 INFO:teuthology.orchestra.run.trial153.stdout: librados2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 3.3 M 2026-02-20T11:53:53.488 INFO:teuthology.orchestra.run.trial153.stdout: librbd1 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.7 M 2026-02-20T11:53:53.488 INFO:teuthology.orchestra.run.trial153.stdout:Installing dependencies: 2026-02-20T11:53:53.488 INFO:teuthology.orchestra.run.trial153.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: ceph-base x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 5.1 M 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: ceph-common x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 21 M 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: ceph-selinux x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 25 k 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 860 k 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-20T11:53:53.489 INFO:teuthology.orchestra.run.trial153.stdout: libradosstriper1 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 263 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: librgw2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 5.6 M 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: python3-ceph-argparse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 45 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: python3-ceph-common x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 185 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: python3-cephfs x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 204 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: python3-rados x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 324 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: python3-rbd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 305 k 2026-02-20T11:53:53.490 INFO:teuthology.orchestra.run.trial153.stdout: python3-rgw x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 100 k 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout:Installing weak dependencies: 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-daemon x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 37 k 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 26 k 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:53:53.491 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:53:53.492 INFO:teuthology.orchestra.run.trial153.stdout:Install 35 Packages 2026-02-20T11:53:53.492 INFO:teuthology.orchestra.run.trial153.stdout:Upgrade 2 Packages 2026-02-20T11:53:53.492 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:53:53.492 INFO:teuthology.orchestra.run.trial183.stdout:(23/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 15 MB/s | 292 kB 00:00 2026-02-20T11:53:53.493 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 61 M 2026-02-20T11:53:53.493 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:53:53.494 INFO:teuthology.orchestra.run.trial183.stdout:(24/37): qatlib-25.08.0-2.el9.x86_64.rpm 23 MB/s | 240 kB 00:00 2026-02-20T11:53:53.499 INFO:teuthology.orchestra.run.trial183.stdout:(25/37): qatlib-service-25.08.0-2.el9.x86_64.rp 4.1 MB/s | 37 kB 00:00 2026-02-20T11:53:53.506 INFO:teuthology.orchestra.run.trial183.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 5.6 MB/s | 66 kB 00:00 2026-02-20T11:53:53.532 INFO:teuthology.orchestra.run.trial183.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 9.0 MB/s | 308 kB 00:00 2026-02-20T11:53:53.536 INFO:teuthology.orchestra.run.trial183.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 6.7 MB/s | 25 kB 00:00 2026-02-20T11:53:53.540 INFO:teuthology.orchestra.run.trial183.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 13 MB/s | 49 kB 00:00 2026-02-20T11:53:53.543 INFO:teuthology.orchestra.run.trial183.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 18 MB/s | 67 kB 00:00 2026-02-20T11:53:53.554 INFO:teuthology.orchestra.run.trial183.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 77 MB/s | 838 kB 00:00 2026-02-20T11:53:53.559 INFO:teuthology.orchestra.run.trial183.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 41 MB/s | 191 kB 00:00 2026-02-20T11:53:53.572 INFO:teuthology.orchestra.run.trial183.stdout:(33/37): thrift-0.15.0-4.el9.x86_64.rpm 125 MB/s | 1.6 MB 00:00 2026-02-20T11:53:53.787 INFO:teuthology.orchestra.run.trial183.stdout:(34/37): libarrow-9.0.0-15.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2026-02-20T11:53:54.307 INFO:teuthology.orchestra.run.trial183.stdout:(35/37): librbd1-20.3.0-5357.g0c3148ca.el9.x86_ 5.2 MB/s | 2.7 MB 00:00 2026-02-20T11:53:54.449 INFO:teuthology.orchestra.run.trial183.stdout:(36/37): librados2-20.3.0-5357.g0c3148ca.el9.x8 3.7 MB/s | 3.3 MB 00:00 2026-02-20T11:53:55.124 INFO:teuthology.orchestra.run.trial153.stdout:(1/37): ceph-base-20.3.0-5357.g0c3148ca.el9.x86 4.5 MB/s | 5.1 MB 00:01 2026-02-20T11:53:55.189 INFO:teuthology.orchestra.run.trial153.stdout:(2/37): ceph-selinux-20.3.0-5357.g0c3148ca.el9. 388 kB/s | 25 kB 00:00 2026-02-20T11:53:55.245 INFO:teuthology.orchestra.run.trial153.stdout:(3/37): libcephfs-daemon-20.3.0-5357.g0c3148ca. 658 kB/s | 37 kB 00:00 2026-02-20T11:53:55.301 INFO:teuthology.orchestra.run.trial153.stdout:(4/37): libcephfs-proxy2-20.3.0-5357.g0c3148ca. 458 kB/s | 26 kB 00:00 2026-02-20T11:53:55.428 INFO:teuthology.orchestra.run.trial153.stdout:(5/37): libcephfs2-20.3.0-5357.g0c3148ca.el9.x8 6.6 MB/s | 860 kB 00:00 2026-02-20T11:53:55.490 INFO:teuthology.orchestra.run.trial153.stdout:(6/37): libradosstriper1-20.3.0-5357.g0c3148ca. 4.2 MB/s | 263 kB 00:00 2026-02-20T11:53:55.522 INFO:teuthology.orchestra.run.trial153.stdout:(7/37): ceph-radosgw-20.3.0-5357.g0c3148ca.el9. 7.3 MB/s | 11 MB 00:01 2026-02-20T11:53:55.577 INFO:teuthology.orchestra.run.trial153.stdout:(8/37): python3-ceph-argparse-20.3.0-5357.g0c31 821 kB/s | 45 kB 00:00 2026-02-20T11:53:55.635 INFO:teuthology.orchestra.run.trial153.stdout:(9/37): python3-ceph-common-20.3.0-5357.g0c3148 3.1 MB/s | 185 kB 00:00 2026-02-20T11:53:55.693 INFO:teuthology.orchestra.run.trial153.stdout:(10/37): python3-cephfs-20.3.0-5357.g0c3148ca.e 3.4 MB/s | 204 kB 00:00 2026-02-20T11:53:55.755 INFO:teuthology.orchestra.run.trial153.stdout:(11/37): python3-rados-20.3.0-5357.g0c3148ca.el 5.2 MB/s | 324 kB 00:00 2026-02-20T11:53:55.817 INFO:teuthology.orchestra.run.trial153.stdout:(12/37): python3-rbd-20.3.0-5357.g0c3148ca.el9. 4.8 MB/s | 305 kB 00:00 2026-02-20T11:53:55.874 INFO:teuthology.orchestra.run.trial153.stdout:(13/37): python3-rgw-20.3.0-5357.g0c3148ca.el9. 1.7 MB/s | 100 kB 00:00 2026-02-20T11:53:56.060 INFO:teuthology.orchestra.run.trial153.stdout:(14/37): librgw2-20.3.0-5357.g0c3148ca.el9.x86_ 9.9 MB/s | 5.6 MB 00:00 2026-02-20T11:53:56.198 INFO:teuthology.orchestra.run.trial183.stdout:(37/37): ceph-common-20.3.0-5357.g0c3148ca.el9. 3.3 MB/s | 21 MB 00:06 2026-02-20T11:53:56.201 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:53:56.201 INFO:teuthology.orchestra.run.trial183.stdout:Total 9.4 MB/s | 61 MB 00:06 2026-02-20T11:53:56.299 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:53:56.319 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:53:56.319 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:53:56.435 INFO:teuthology.orchestra.run.trial153.stdout:(15/37): mailcap-2.1.49-5.el9.noarch.rpm 59 kB/s | 33 kB 00:00 2026-02-20T11:53:56.570 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:53:56.570 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:53:56.860 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:53:56.996 INFO:teuthology.orchestra.run.trial153.stdout:(16/37): nvme-cli-2.16-1.el9.x86_64.rpm 1.2 MB/s | 1.2 MB 00:00 2026-02-20T11:53:57.011 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-20T11:53:57.016 INFO:teuthology.orchestra.run.trial183.stdout: Upgrading : librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:53:57.064 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:53:57.065 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:53:57.078 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:53:57.080 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_6 4/39 2026-02-20T11:53:57.081 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:53:57.091 INFO:teuthology.orchestra.run.trial153.stdout:(17/37): pciutils-3.7.0-7.el9.x86_64.rpm 142 kB/s | 93 kB 00:00 2026-02-20T11:53:57.107 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:53:57.110 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 6/39 2026-02-20T11:53:57.117 INFO:teuthology.orchestra.run.trial183.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-20T11:53:57.121 INFO:teuthology.orchestra.run.trial153.stdout:(18/37): boost-program-options-1.75.0-13.el9.x8 837 kB/s | 104 kB 00:00 2026-02-20T11:53:57.121 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-20T11:53:57.124 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-20T11:53:57.128 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-20T11:53:57.139 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9. 11/39 2026-02-20T11:53:57.152 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/39 2026-02-20T11:53:57.153 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 13/39 2026-02-20T11:53:57.174 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 13/39 2026-02-20T11:53:57.177 INFO:teuthology.orchestra.run.trial183.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-20T11:53:57.203 INFO:teuthology.orchestra.run.trial153.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2026-02-20T11:53:57.208 INFO:teuthology.orchestra.run.trial183.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-20T11:53:57.214 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-20T11:53:57.220 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-20T11:53:57.222 INFO:teuthology.orchestra.run.trial183.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-20T11:53:57.227 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-20T11:53:57.230 INFO:teuthology.orchestra.run.trial153.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.6 MB/s | 45 kB 00:00 2026-02-20T11:53:57.232 INFO:teuthology.orchestra.run.trial153.stdout:(21/37): libnbd-1.20.3-4.el9.x86_64.rpm 1.1 MB/s | 164 kB 00:00 2026-02-20T11:53:57.292 INFO:teuthology.orchestra.run.trial153.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 292 kB 00:00 2026-02-20T11:53:57.302 INFO:teuthology.orchestra.run.trial153.stdout:(23/37): librdkafka-1.6.1-102.el9.x86_64.rpm 9.0 MB/s | 662 kB 00:00 2026-02-20T11:53:57.327 INFO:teuthology.orchestra.run.trial153.stdout:(24/37): qatlib-25.08.0-2.el9.x86_64.rpm 6.7 MB/s | 240 kB 00:00 2026-02-20T11:53:57.327 INFO:teuthology.orchestra.run.trial153.stdout:(25/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.4 MB/s | 37 kB 00:00 2026-02-20T11:53:57.350 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-20T11:53:57.351 INFO:teuthology.orchestra.run.trial183.stdout: Upgrading : librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 21/39 2026-02-20T11:53:57.354 INFO:teuthology.orchestra.run.trial153.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.4 MB/s | 66 kB 00:00 2026-02-20T11:53:57.376 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 21/39 2026-02-20T11:53:57.379 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 22/39 2026-02-20T11:53:57.385 INFO:teuthology.orchestra.run.trial183.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-20T11:53:57.495 INFO:teuthology.orchestra.run.trial153.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.8 MB/s | 308 kB 00:00 2026-02-20T11:53:57.509 INFO:teuthology.orchestra.run.trial153.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 1.7 MB/s | 25 kB 00:00 2026-02-20T11:53:57.524 INFO:teuthology.orchestra.run.trial153.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 3.2 MB/s | 49 kB 00:00 2026-02-20T11:53:57.539 INFO:teuthology.orchestra.run.trial153.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 4.3 MB/s | 67 kB 00:00 2026-02-20T11:53:57.553 INFO:teuthology.orchestra.run.trial153.stdout:(31/37): libarrow-9.0.0-15.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2026-02-20T11:53:57.570 INFO:teuthology.orchestra.run.trial153.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 11 MB/s | 191 kB 00:00 2026-02-20T11:53:57.581 INFO:teuthology.orchestra.run.trial153.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 20 MB/s | 838 kB 00:00 2026-02-20T11:53:57.590 INFO:teuthology.orchestra.run.trial153.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 79 MB/s | 1.6 MB 00:00 2026-02-20T11:53:57.646 INFO:teuthology.orchestra.run.trial183.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-20T11:53:57.647 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 25/39 2026-02-20T11:53:57.663 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 25/39 2026-02-20T11:53:57.667 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 26/39 2026-02-20T11:53:57.673 INFO:teuthology.orchestra.run.trial183.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-20T11:53:57.676 INFO:teuthology.orchestra.run.trial183.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-20T11:53:57.677 INFO:teuthology.orchestra.run.trial183.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-20T11:53:57.694 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-20T11:53:57.727 INFO:teuthology.orchestra.run.trial153.stdout:(35/37): ceph-common-20.3.0-5357.g0c3148ca.el9. 5.6 MB/s | 21 MB 00:03 2026-02-20T11:53:57.806 INFO:teuthology.orchestra.run.trial183.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-20T11:53:57.808 INFO:teuthology.orchestra.run.trial183.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-20T11:53:58.034 INFO:teuthology.orchestra.run.trial153.stdout:(36/37): librbd1-20.3.0-5357.g0c3148ca.el9.x86_ 6.1 MB/s | 2.7 MB 00:00 2026-02-20T11:53:58.094 INFO:teuthology.orchestra.run.trial153.stdout:(37/37): librados2-20.3.0-5357.g0c3148ca.el9.x8 6.4 MB/s | 3.3 MB 00:00 2026-02-20T11:53:58.095 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:53:58.095 INFO:teuthology.orchestra.run.trial153.stdout:Total 13 MB/s | 61 MB 00:04 2026-02-20T11:53:58.102 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-20T11:53:58.102 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-20T11:53:58.102 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:53:58.147 INFO:teuthology.orchestra.run.trial183.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-20T11:53:58.149 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x8 33/39 2026-02-20T11:53:58.202 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:53:58.222 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:53:58.222 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:53:58.474 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:53:58.475 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:53:58.766 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:53:58.909 INFO:teuthology.orchestra.run.trial153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-20T11:53:58.913 INFO:teuthology.orchestra.run.trial153.stdout: Upgrading : librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:53:58.962 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:53:58.963 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:53:58.975 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:53:58.978 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_6 4/39 2026-02-20T11:53:58.978 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:53:59.006 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:53:59.008 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 6/39 2026-02-20T11:53:59.016 INFO:teuthology.orchestra.run.trial153.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-20T11:53:59.018 INFO:teuthology.orchestra.run.trial153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-20T11:53:59.021 INFO:teuthology.orchestra.run.trial153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-20T11:53:59.025 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-20T11:53:59.036 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9. 11/39 2026-02-20T11:53:59.049 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/39 2026-02-20T11:53:59.050 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 13/39 2026-02-20T11:53:59.071 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 13/39 2026-02-20T11:53:59.075 INFO:teuthology.orchestra.run.trial153.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-20T11:53:59.105 INFO:teuthology.orchestra.run.trial153.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-20T11:53:59.109 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-20T11:53:59.115 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-20T11:53:59.117 INFO:teuthology.orchestra.run.trial153.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-20T11:53:59.122 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-20T11:53:59.152 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:53:59.154 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:53:59.237 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-20T11:53:59.238 INFO:teuthology.orchestra.run.trial153.stdout: Upgrading : librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 21/39 2026-02-20T11:53:59.264 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 21/39 2026-02-20T11:53:59.266 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 22/39 2026-02-20T11:53:59.274 INFO:teuthology.orchestra.run.trial153.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-20T11:53:59.429 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:53:59.432 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 35/39 2026-02-20T11:53:59.467 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 35/39 2026-02-20T11:53:59.467 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-20T11:53:59.467 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-20T11:53:59.467 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:53:59.471 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:53:59.526 INFO:teuthology.orchestra.run.trial153.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-20T11:53:59.527 INFO:teuthology.orchestra.run.trial153.stdout: Installing : librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 25/39 2026-02-20T11:53:59.543 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 25/39 2026-02-20T11:53:59.547 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 26/39 2026-02-20T11:53:59.552 INFO:teuthology.orchestra.run.trial153.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-20T11:53:59.555 INFO:teuthology.orchestra.run.trial153.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-20T11:53:59.556 INFO:teuthology.orchestra.run.trial153.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-20T11:53:59.573 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-20T11:53:59.688 INFO:teuthology.orchestra.run.trial153.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-20T11:53:59.690 INFO:teuthology.orchestra.run.trial153.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-20T11:53:59.975 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-20T11:53:59.975 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-20T11:53:59.976 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:00.021 INFO:teuthology.orchestra.run.trial153.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-20T11:54:00.023 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x8 33/39 2026-02-20T11:54:00.970 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:54:00.972 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:54:01.236 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 34/39 2026-02-20T11:54:01.240 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 35/39 2026-02-20T11:54:01.283 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 35/39 2026-02-20T11:54:01.283 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-20T11:54:01.283 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-20T11:54:01.283 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:01.286 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:54:04.016 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:54:04.016 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /sys 2026-02-20T11:54:04.016 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /proc 2026-02-20T11:54:04.016 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /mnt 2026-02-20T11:54:04.016 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /var/tmp 2026-02-20T11:54:04.017 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /home 2026-02-20T11:54:04.017 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /root 2026-02-20T11:54:04.017 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /tmp 2026-02-20T11:54:04.017 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:54:04.546 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 37/39 2026-02-20T11:54:04.568 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 37/39 2026-02-20T11:54:04.569 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:54:04.569 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-20T11:54:04.569 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-20T11:54:04.569 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-20T11:54:04.569 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:54:04.571 INFO:teuthology.orchestra.run.trial183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-20T11:54:04.584 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-20T11:54:04.584 INFO:teuthology.orchestra.run.trial183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:54:04.640 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:54:04.640 INFO:teuthology.orchestra.run.trial183.stdout:Creating group 'qat' with GID 992. 2026-02-20T11:54:04.640 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:54:04.936 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/39 2026-02-20T11:54:04.936 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:54:04.936 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:54:04.936 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 4/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 6/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 7/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 8/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 9/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9. 10/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x8 11/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 13/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 14/39 2026-02-20T11:54:04.937 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 15/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-20T11:54:04.938 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-20T11:54:04.939 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-20T11:54:04.939 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-20T11:54:04.939 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-20T11:54:04.939 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-20T11:54:04.939 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-20T11:54:04.939 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-20T11:54:04.939 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-20T11:54:04.939 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-20T11:54:04.939 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-20T11:54:04.940 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:54:04.940 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-20T11:54:04.940 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 38/39 2026-02-20T11:54:04.999 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:54:04.999 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:54:04.999 INFO:teuthology.orchestra.run.trial183.stdout:Upgraded: 2026-02-20T11:54:04.999 INFO:teuthology.orchestra.run.trial183.stdout: librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:04.999 INFO:teuthology.orchestra.run.trial183.stdout: librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.000 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:54:05.000 INFO:teuthology.orchestra.run.trial183.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-20T11:54:05.000 INFO:teuthology.orchestra.run.trial183.stdout: ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.000 INFO:teuthology.orchestra.run.trial183.stdout: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.000 INFO:teuthology.orchestra.run.trial183.stdout: ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.000 INFO:teuthology.orchestra.run.trial183.stdout: ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.000 INFO:teuthology.orchestra.run.trial183.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-20T11:54:05.001 INFO:teuthology.orchestra.run.trial183.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-20T11:54:05.001 INFO:teuthology.orchestra.run.trial183.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-20T11:54:05.001 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.001 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.001 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.001 INFO:teuthology.orchestra.run.trial183.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-20T11:54:05.002 INFO:teuthology.orchestra.run.trial183.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-20T11:54:05.002 INFO:teuthology.orchestra.run.trial183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-20T11:54:05.002 INFO:teuthology.orchestra.run.trial183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-20T11:54:05.002 INFO:teuthology.orchestra.run.trial183.stdout: libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.002 INFO:teuthology.orchestra.run.trial183.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-20T11:54:05.002 INFO:teuthology.orchestra.run.trial183.stdout: librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.002 INFO:teuthology.orchestra.run.trial183.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-20T11:54:05.003 INFO:teuthology.orchestra.run.trial183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-20T11:54:05.003 INFO:teuthology.orchestra.run.trial183.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-20T11:54:05.003 INFO:teuthology.orchestra.run.trial183.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-20T11:54:05.003 INFO:teuthology.orchestra.run.trial183.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-20T11:54:05.003 INFO:teuthology.orchestra.run.trial183.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-20T11:54:05.003 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.004 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.004 INFO:teuthology.orchestra.run.trial183.stdout: python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.004 INFO:teuthology.orchestra.run.trial183.stdout: python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.004 INFO:teuthology.orchestra.run.trial183.stdout: python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.004 INFO:teuthology.orchestra.run.trial183.stdout: python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:05.004 INFO:teuthology.orchestra.run.trial183.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-20T11:54:05.004 INFO:teuthology.orchestra.run.trial183.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-20T11:54:05.005 INFO:teuthology.orchestra.run.trial183.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-20T11:54:05.005 INFO:teuthology.orchestra.run.trial183.stdout: re2-1:20211101-20.el9.x86_64 2026-02-20T11:54:05.005 INFO:teuthology.orchestra.run.trial183.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-20T11:54:05.005 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:54:05.005 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:54:05.042 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-test 2026-02-20T11:54:05.350 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:17 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:54:05.434 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:54:05.434 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repository Size 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: ceph-test x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 127 M 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: libcephsqlite x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 153 k 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout:Install 12 Packages 2026-02-20T11:54:05.435 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:54:05.436 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 132 M 2026-02-20T11:54:05.436 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 989 M 2026-02-20T11:54:05.436 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:54:05.824 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:54:05.824 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /sys 2026-02-20T11:54:05.824 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /proc 2026-02-20T11:54:05.824 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /mnt 2026-02-20T11:54:05.824 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /var/tmp 2026-02-20T11:54:05.825 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /home 2026-02-20T11:54:05.825 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /root 2026-02-20T11:54:05.825 INFO:teuthology.orchestra.run.trial153.stdout:skipping the directory /tmp 2026-02-20T11:54:05.825 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:06.090 INFO:teuthology.orchestra.run.trial183.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 9.2 MB/s | 2.1 MB 00:00 2026-02-20T11:54:06.112 INFO:teuthology.orchestra.run.trial183.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 1.0 MB/s | 23 kB 00:00 2026-02-20T11:54:06.134 INFO:teuthology.orchestra.run.trial183.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 4.1 MB/s | 94 kB 00:00 2026-02-20T11:54:06.257 INFO:teuthology.orchestra.run.trial183.stdout:(4/12): libcephsqlite-20.3.0-5357.g0c3148ca.el9 388 kB/s | 153 kB 00:00 2026-02-20T11:54:06.315 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 37/39 2026-02-20T11:54:06.338 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 37/39 2026-02-20T11:54:06.338 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:54:06.339 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-20T11:54:06.339 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-20T11:54:06.339 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-20T11:54:06.339 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:06.340 INFO:teuthology.orchestra.run.trial153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-20T11:54:06.353 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-20T11:54:06.353 INFO:teuthology.orchestra.run.trial153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:54:06.409 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:54:06.409 INFO:teuthology.orchestra.run.trial153.stdout:Creating group 'qat' with GID 992. 2026-02-20T11:54:06.410 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:06.711 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/39 2026-02-20T11:54:06.712 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/39 2026-02-20T11:54:06.712 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/39 2026-02-20T11:54:06.712 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 4/39 2026-02-20T11:54:06.712 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_6 5/39 2026-02-20T11:54:06.712 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_6 6/39 2026-02-20T11:54:06.712 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 7/39 2026-02-20T11:54:06.712 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_6 8/39 2026-02-20T11:54:06.712 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 9/39 2026-02-20T11:54:06.712 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9. 10/39 2026-02-20T11:54:06.712 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x8 11/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 13/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 14/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 15/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-20T11:54:06.713 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 36/39 2026-02-20T11:54:06.714 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-20T11:54:06.715 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 38/39 2026-02-20T11:54:06.774 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-20T11:54:06.775 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:06.775 INFO:teuthology.orchestra.run.trial153.stdout:Upgraded: 2026-02-20T11:54:06.775 INFO:teuthology.orchestra.run.trial153.stdout: librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.775 INFO:teuthology.orchestra.run.trial153.stdout: librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.775 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:54:06.775 INFO:teuthology.orchestra.run.trial153.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-20T11:54:06.775 INFO:teuthology.orchestra.run.trial153.stdout: ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.775 INFO:teuthology.orchestra.run.trial153.stdout: ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.775 INFO:teuthology.orchestra.run.trial153.stdout: ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.775 INFO:teuthology.orchestra.run.trial153.stdout: ceph-selinux-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-daemon-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-proxy2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: libradosstriper1-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.776 INFO:teuthology.orchestra.run.trial153.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: librgw2-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: python3-ceph-argparse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: python3-ceph-common-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.777 INFO:teuthology.orchestra.run.trial153.stdout: python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:06.778 INFO:teuthology.orchestra.run.trial153.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-20T11:54:06.778 INFO:teuthology.orchestra.run.trial153.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-20T11:54:06.778 INFO:teuthology.orchestra.run.trial153.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-20T11:54:06.778 INFO:teuthology.orchestra.run.trial153.stdout: re2-1:20211101-20.el9.x86_64 2026-02-20T11:54:06.778 INFO:teuthology.orchestra.run.trial153.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-20T11:54:06.778 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:06.778 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:54:06.816 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-test 2026-02-20T11:54:07.130 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:15 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:54:07.181 INFO:teuthology.orchestra.run.trial183.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 190 kB/s | 176 kB 00:00 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repository Size 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout: ceph-test x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 127 M 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout:Installing dependencies: 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout: libcephsqlite x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 153 k 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-20T11:54:07.214 INFO:teuthology.orchestra.run.trial153.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout:Install 12 Packages 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 132 M 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 989 M 2026-02-20T11:54:07.215 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:54:07.353 INFO:teuthology.orchestra.run.trial183.stdout:(6/12): libasan8-15.2.1-7.el9.x86_64.rpm 431 kB/s | 525 kB 00:01 2026-02-20T11:54:07.443 INFO:teuthology.orchestra.run.trial183.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 890 kB/s | 233 kB 00:00 2026-02-20T11:54:07.618 INFO:teuthology.orchestra.run.trial183.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2026-02-20T11:54:07.620 INFO:teuthology.orchestra.run.trial183.stdout:(9/12): socat-1.7.4.1-8.el9.x86_64.rpm 1.7 MB/s | 303 kB 00:00 2026-02-20T11:54:07.773 INFO:teuthology.orchestra.run.trial183.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 798 kB/s | 122 kB 00:00 2026-02-20T11:54:07.795 INFO:teuthology.orchestra.run.trial183.stdout:(11/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 360 kB/s | 64 kB 00:00 2026-02-20T11:54:07.872 INFO:teuthology.orchestra.run.trial153.stdout:(1/12): libcephsqlite-20.3.0-5357.g0c3148ca.el9 389 kB/s | 153 kB 00:00 2026-02-20T11:54:08.316 INFO:teuthology.orchestra.run.trial153.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 51 kB/s | 23 kB 00:00 2026-02-20T11:54:08.494 INFO:teuthology.orchestra.run.trial153.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 529 kB/s | 94 kB 00:00 2026-02-20T11:54:08.598 INFO:teuthology.orchestra.run.trial153.stdout:(4/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 1.9 MB/s | 2.1 MB 00:01 2026-02-20T11:54:09.068 INFO:teuthology.orchestra.run.trial153.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 375 kB/s | 176 kB 00:00 2026-02-20T11:54:09.154 INFO:teuthology.orchestra.run.trial153.stdout:(6/12): libasan8-15.2.1-7.el9.x86_64.rpm 795 kB/s | 525 kB 00:00 2026-02-20T11:54:09.233 INFO:teuthology.orchestra.run.trial153.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 1.4 MB/s | 233 kB 00:00 2026-02-20T11:54:09.412 INFO:teuthology.orchestra.run.trial153.stdout:(8/12): protobuf-3.14.0-17.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2026-02-20T11:54:09.437 INFO:teuthology.orchestra.run.trial153.stdout:(9/12): socat-1.7.4.1-8.el9.x86_64.rpm 1.4 MB/s | 303 kB 00:00 2026-02-20T11:54:09.546 INFO:teuthology.orchestra.run.trial153.stdout:(10/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 476 kB/s | 64 kB 00:00 2026-02-20T11:54:09.582 INFO:teuthology.orchestra.run.trial153.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 839 kB/s | 122 kB 00:00 2026-02-20T11:54:17.459 INFO:teuthology.orchestra.run.trial153.stdout:(12/12): ceph-test-20.3.0-5357.g0c3148ca.el9.x8 13 MB/s | 127 MB 00:09 2026-02-20T11:54:17.462 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:54:17.462 INFO:teuthology.orchestra.run.trial153.stdout:Total 13 MB/s | 132 MB 00:10 2026-02-20T11:54:17.546 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:54:17.556 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:54:17.557 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:54:17.984 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:54:17.984 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:54:18.438 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:54:18.447 INFO:teuthology.orchestra.run.trial153.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-20T11:54:18.456 INFO:teuthology.orchestra.run.trial153.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-20T11:54:18.461 INFO:teuthology.orchestra.run.trial153.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-20T11:54:18.465 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-20T11:54:18.467 INFO:teuthology.orchestra.run.trial153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-20T11:54:18.472 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-20T11:54:18.472 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-20T11:54:18.491 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-20T11:54:18.493 INFO:teuthology.orchestra.run.trial153.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-20T11:54:18.503 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-20T11:54:18.511 INFO:teuthology.orchestra.run.trial153.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-20T11:54:18.512 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 11/12 2026-02-20T11:54:25.674 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 11/12 2026-02-20T11:54:25.692 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/12 2026-02-20T11:54:26.499 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-20T11:54:26.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-20T11:54:26.553 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-20T11:54:26.553 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:26.553 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:54:26.553 INFO:teuthology.orchestra.run.trial153.stdout: ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-20T11:54:26.554 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:26.555 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:54:26.594 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph 2026-02-20T11:54:26.895 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:34 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:54:26.982 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:54:26.984 INFO:teuthology.orchestra.run.trial153.stdout:======================================================================================= 2026-02-20T11:54:26.984 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repository Size 2026-02-20T11:54:26.985 INFO:teuthology.orchestra.run.trial153.stdout:======================================================================================= 2026-02-20T11:54:26.985 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:54:26.985 INFO:teuthology.orchestra.run.trial153.stdout: ceph x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 6.5 k 2026-02-20T11:54:26.985 INFO:teuthology.orchestra.run.trial153.stdout:Installing dependencies: 2026-02-20T11:54:26.985 INFO:teuthology.orchestra.run.trial153.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-20T11:54:26.985 INFO:teuthology.orchestra.run.trial153.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 57 k 2026-02-20T11:54:26.985 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mds x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.0 M 2026-02-20T11:54:26.985 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 937 k 2026-02-20T11:54:26.985 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 297 k 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mon x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 4.9 M 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 163 k 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-classic x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 15 M 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-crimson x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 88 M 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 17 k 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: cephadm noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 1.0 M 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-20T11:54:26.986 INFO:teuthology.orchestra.run.trial153.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-20T11:54:26.987 INFO:teuthology.orchestra.run.trial153.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-20T11:54:26.988 INFO:teuthology.orchestra.run.trial153.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-20T11:54:26.989 INFO:teuthology.orchestra.run.trial153.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-20T11:54:26.990 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-20T11:54:26.991 INFO:teuthology.orchestra.run.trial153.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout:Installing weak dependencies: 2026-02-20T11:54:26.992 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 184 k 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 51 M 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 7.4 M 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 22 k 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-rook noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 50 k 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout: ceph-volume noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 301 k 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout:======================================================================================= 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout:Install 96 Packages 2026-02-20T11:54:26.993 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:54:26.994 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 226 M 2026-02-20T11:54:26.995 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 1.5 G 2026-02-20T11:54:26.995 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:54:27.555 INFO:teuthology.orchestra.run.trial153.stdout:(1/96): ceph-20.3.0-5357.g0c3148ca.el9.x86_64.r 24 kB/s | 6.5 kB 00:00 2026-02-20T11:54:28.084 INFO:teuthology.orchestra.run.trial153.stdout:(2/96): ceph-mgr-20.3.0-5357.g0c3148ca.el9.x86_ 1.1 MB/s | 937 kB 00:00 2026-02-20T11:54:28.148 INFO:teuthology.orchestra.run.trial153.stdout:(3/96): ceph-osd-20.3.0-5357.g0c3148ca.el9.x86_ 2.5 MB/s | 163 kB 00:00 2026-02-20T11:54:28.203 INFO:teuthology.orchestra.run.trial153.stdout:(4/96): ceph-mds-20.3.0-5357.g0c3148ca.el9.x86_ 2.2 MB/s | 2.0 MB 00:00 2026-02-20T11:54:30.506 INFO:teuthology.orchestra.run.trial153.stdout:(5/96): ceph-mon-20.3.0-5357.g0c3148ca.el9.x86_ 1.7 MB/s | 4.9 MB 00:02 2026-02-20T11:54:30.572 INFO:teuthology.orchestra.run.trial153.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5357.g0c 868 kB/s | 57 kB 00:00 2026-02-20T11:54:30.711 INFO:teuthology.orchestra.run.trial153.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5357.g0c3148ca. 1.3 MB/s | 184 kB 00:00 2026-02-20T11:54:34.319 INFO:teuthology.orchestra.run.trial153.stdout:(8/96): ceph-osd-classic-20.3.0-5357.g0c3148ca. 2.4 MB/s | 15 MB 00:06 2026-02-20T11:54:39.272 INFO:teuthology.orchestra.run.trial153.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 1.5 MB/s | 7.4 MB 00:04 2026-02-20T11:54:39.340 INFO:teuthology.orchestra.run.trial153.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5357.g0c3148 326 kB/s | 22 kB 00:00 2026-02-20T11:54:39.548 INFO:teuthology.orchestra.run.trial153.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5357.g0c3 1.4 MB/s | 297 kB 00:00 2026-02-20T11:54:39.608 INFO:teuthology.orchestra.run.trial153.stdout:(12/96): ceph-mgr-rook-20.3.0-5357.g0c3148ca.el 846 kB/s | 50 kB 00:00 2026-02-20T11:54:39.668 INFO:teuthology.orchestra.run.trial153.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5357.g0c 288 kB/s | 17 kB 00:00 2026-02-20T11:54:39.854 INFO:teuthology.orchestra.run.trial153.stdout:(14/96): ceph-volume-20.3.0-5357.g0c3148ca.el9. 1.6 MB/s | 301 kB 00:00 2026-02-20T11:54:40.463 INFO:teuthology.orchestra.run.trial153.stdout:(15/96): cephadm-20.3.0-5357.g0c3148ca.el9.noar 1.7 MB/s | 1.0 MB 00:00 2026-02-20T11:54:41.027 INFO:teuthology.orchestra.run.trial153.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 622 kB/s | 351 kB 00:00 2026-02-20T11:54:41.114 INFO:teuthology.orchestra.run.trial153.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 461 kB/s | 40 kB 00:00 2026-02-20T11:54:41.207 INFO:teuthology.orchestra.run.trial153.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 779 kB/s | 72 kB 00:00 2026-02-20T11:54:41.420 INFO:teuthology.orchestra.run.trial153.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 3.6 MB/s | 794 kB 00:00 2026-02-20T11:54:41.512 INFO:teuthology.orchestra.run.trial153.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 2.0 MB/s | 184 kB 00:00 2026-02-20T11:54:41.610 INFO:teuthology.orchestra.run.trial153.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2026-02-20T11:54:41.767 INFO:teuthology.orchestra.run.trial153.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 7.9 MB/s | 1.2 MB 00:00 2026-02-20T11:54:41.854 INFO:teuthology.orchestra.run.trial153.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2026-02-20T11:54:41.941 INFO:teuthology.orchestra.run.trial153.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 135 kB 00:00 2026-02-20T11:54:42.029 INFO:teuthology.orchestra.run.trial153.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 1.7 MB/s | 150 kB 00:00 2026-02-20T11:54:42.116 INFO:teuthology.orchestra.run.trial153.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 1.4 MB/s | 126 kB 00:00 2026-02-20T11:54:42.206 INFO:teuthology.orchestra.run.trial153.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 2.3 MB/s | 218 kB 00:00 2026-02-20T11:54:42.295 INFO:teuthology.orchestra.run.trial153.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 2.0 MB/s | 182 kB 00:00 2026-02-20T11:54:42.388 INFO:teuthology.orchestra.run.trial153.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 2.8 MB/s | 266 kB 00:00 2026-02-20T11:54:42.398 INFO:teuthology.orchestra.run.trial153.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 2.7 MB/s | 30 kB 00:00 2026-02-20T11:54:42.426 INFO:teuthology.orchestra.run.trial153.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 109 MB/s | 3.0 MB 00:00 2026-02-20T11:54:42.430 INFO:teuthology.orchestra.run.trial153.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 3.6 MB/s | 15 kB 00:00 2026-02-20T11:54:42.436 INFO:teuthology.orchestra.run.trial153.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 41 MB/s | 246 kB 00:00 2026-02-20T11:54:42.442 INFO:teuthology.orchestra.run.trial153.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 31 MB/s | 188 kB 00:00 2026-02-20T11:54:42.447 INFO:teuthology.orchestra.run.trial153.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 9.8 MB/s | 42 kB 00:00 2026-02-20T11:54:42.468 INFO:teuthology.orchestra.run.trial153.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 246 MB/s | 5.3 MB 00:00 2026-02-20T11:54:42.493 INFO:teuthology.orchestra.run.trial153.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 238 MB/s | 6.0 MB 00:00 2026-02-20T11:54:42.501 INFO:teuthology.orchestra.run.trial153.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 33 MB/s | 244 kB 00:00 2026-02-20T11:54:42.508 INFO:teuthology.orchestra.run.trial153.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 35 MB/s | 249 kB 00:00 2026-02-20T11:54:42.513 INFO:teuthology.orchestra.run.trial153.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 9.3 MB/s | 48 kB 00:00 2026-02-20T11:54:42.519 INFO:teuthology.orchestra.run.trial153.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 30 MB/s | 177 kB 00:00 2026-02-20T11:54:42.528 INFO:teuthology.orchestra.run.trial153.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 126 MB/s | 1.2 MB 00:00 2026-02-20T11:54:42.533 INFO:teuthology.orchestra.run.trial153.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 6.9 MB/s | 35 kB 00:00 2026-02-20T11:54:42.558 INFO:teuthology.orchestra.run.trial153.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 256 MB/s | 6.1 MB 00:00 2026-02-20T11:54:42.565 INFO:teuthology.orchestra.run.trial153.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 56 MB/s | 442 kB 00:00 2026-02-20T11:54:42.571 INFO:teuthology.orchestra.run.trial153.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 14 MB/s | 77 kB 00:00 2026-02-20T11:54:42.577 INFO:teuthology.orchestra.run.trial153.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 44 MB/s | 267 kB 00:00 2026-02-20T11:54:42.583 INFO:teuthology.orchestra.run.trial153.stdout:(48/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 24 MB/s | 159 kB 00:00 2026-02-20T11:54:42.591 INFO:teuthology.orchestra.run.trial153.stdout:(49/96): python3-pyasn1-modules-0.4.8-6.el9.noa 39 MB/s | 279 kB 00:00 2026-02-20T11:54:42.596 INFO:teuthology.orchestra.run.trial153.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 10 MB/s | 54 kB 00:00 2026-02-20T11:54:42.660 INFO:teuthology.orchestra.run.trial153.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 304 MB/s | 19 MB 00:00 2026-02-20T11:54:42.665 INFO:teuthology.orchestra.run.trial153.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 7.6 MB/s | 42 kB 00:00 2026-02-20T11:54:42.671 INFO:teuthology.orchestra.run.trial153.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 33 MB/s | 189 kB 00:00 2026-02-20T11:54:42.676 INFO:teuthology.orchestra.run.trial153.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 18 MB/s | 90 kB 00:00 2026-02-20T11:54:42.758 INFO:teuthology.orchestra.run.trial153.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 271 kB/s | 22 kB 00:00 2026-02-20T11:54:42.877 INFO:teuthology.orchestra.run.trial153.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 7.1 MB/s | 862 kB 00:00 2026-02-20T11:54:43.013 INFO:teuthology.orchestra.run.trial153.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 4.0 MB/s | 551 kB 00:00 2026-02-20T11:54:43.021 INFO:teuthology.orchestra.run.trial153.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 2.4 MB/s | 19 kB 00:00 2026-02-20T11:54:43.030 INFO:teuthology.orchestra.run.trial153.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 17 MB/s | 151 kB 00:00 2026-02-20T11:54:43.041 INFO:teuthology.orchestra.run.trial153.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 48 MB/s | 548 kB 00:00 2026-02-20T11:54:43.049 INFO:teuthology.orchestra.run.trial153.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 3.6 MB/s | 29 kB 00:00 2026-02-20T11:54:43.057 INFO:teuthology.orchestra.run.trial153.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 7.1 MB/s | 60 kB 00:00 2026-02-20T11:54:43.066 INFO:teuthology.orchestra.run.trial153.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 5.1 MB/s | 43 kB 00:00 2026-02-20T11:54:43.075 INFO:teuthology.orchestra.run.trial153.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 3.6 MB/s | 32 kB 00:00 2026-02-20T11:54:43.083 INFO:teuthology.orchestra.run.trial153.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 1.7 MB/s | 14 kB 00:00 2026-02-20T11:54:43.093 INFO:teuthology.orchestra.run.trial153.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 17 MB/s | 173 kB 00:00 2026-02-20T11:54:43.108 INFO:teuthology.orchestra.run.trial153.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 24 MB/s | 358 kB 00:00 2026-02-20T11:54:43.119 INFO:teuthology.orchestra.run.trial153.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 22 MB/s | 254 kB 00:00 2026-02-20T11:54:43.173 INFO:teuthology.orchestra.run.trial153.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 38 MB/s | 2.0 MB 00:00 2026-02-20T11:54:43.184 INFO:teuthology.orchestra.run.trial153.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 14 MB/s | 144 kB 00:00 2026-02-20T11:54:43.195 INFO:teuthology.orchestra.run.trial153.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 12 MB/s | 139 kB 00:00 2026-02-20T11:54:43.204 INFO:teuthology.orchestra.run.trial153.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 6.1 MB/s | 56 kB 00:00 2026-02-20T11:54:43.211 INFO:teuthology.orchestra.run.trial153.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 1.4 MB/s | 11 kB 00:00 2026-02-20T11:54:43.219 INFO:teuthology.orchestra.run.trial153.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 2.3 MB/s | 18 kB 00:00 2026-02-20T11:54:43.227 INFO:teuthology.orchestra.run.trial153.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 2.9 MB/s | 23 kB 00:00 2026-02-20T11:54:43.235 INFO:teuthology.orchestra.run.trial153.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 2.5 MB/s | 20 kB 00:00 2026-02-20T11:54:43.243 INFO:teuthology.orchestra.run.trial153.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 2.5 MB/s | 19 kB 00:00 2026-02-20T11:54:43.250 INFO:teuthology.orchestra.run.trial153.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 3.3 MB/s | 26 kB 00:00 2026-02-20T11:54:43.263 INFO:teuthology.orchestra.run.trial153.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 81 MB/s | 1.0 MB 00:00 2026-02-20T11:54:43.272 INFO:teuthology.orchestra.run.trial153.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 8.8 MB/s | 79 kB 00:00 2026-02-20T11:54:43.281 INFO:teuthology.orchestra.run.trial153.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 9.1 MB/s | 86 kB 00:00 2026-02-20T11:54:43.290 INFO:teuthology.orchestra.run.trial153.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 6.6 MB/s | 58 kB 00:00 2026-02-20T11:54:43.298 INFO:teuthology.orchestra.run.trial153.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 2.1 MB/s | 16 kB 00:00 2026-02-20T11:54:43.307 INFO:teuthology.orchestra.run.trial153.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 9.6 MB/s | 90 kB 00:00 2026-02-20T11:54:43.316 INFO:teuthology.orchestra.run.trial153.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 3.6 MB/s | 31 kB 00:00 2026-02-20T11:54:43.326 INFO:teuthology.orchestra.run.trial153.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 18 MB/s | 188 kB 00:00 2026-02-20T11:54:43.335 INFO:teuthology.orchestra.run.trial153.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 6.5 MB/s | 59 kB 00:00 2026-02-20T11:54:43.344 INFO:teuthology.orchestra.run.trial153.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 13 MB/s | 125 kB 00:00 2026-02-20T11:54:43.352 INFO:teuthology.orchestra.run.trial153.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 4.4 MB/s | 36 kB 00:00 2026-02-20T11:54:43.361 INFO:teuthology.orchestra.run.trial153.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 10 MB/s | 86 kB 00:00 2026-02-20T11:54:43.370 INFO:teuthology.orchestra.run.trial153.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 10 MB/s | 90 kB 00:00 2026-02-20T11:54:43.379 INFO:teuthology.orchestra.run.trial153.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 5.4 MB/s | 48 kB 00:00 2026-02-20T11:54:43.387 INFO:teuthology.orchestra.run.trial153.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 2.7 MB/s | 22 kB 00:00 2026-02-20T11:54:43.394 INFO:teuthology.orchestra.run.trial153.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 2.6 MB/s | 20 kB 00:00 2026-02-20T11:54:49.222 INFO:teuthology.orchestra.run.trial153.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5357.g0c3148 2.7 MB/s | 51 MB 00:18 2026-02-20T11:55:06.088 INFO:teuthology.orchestra.run.trial183.stdout:(12/12): ceph-test-20.3.0-5357.g0c3148ca.el9.x8 2.1 MB/s | 127 MB 01:00 2026-02-20T11:55:06.091 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:06.091 INFO:teuthology.orchestra.run.trial183.stdout:Total 2.2 MB/s | 132 MB 01:00 2026-02-20T11:55:06.159 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:55:06.171 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:55:06.171 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:55:06.604 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:55:06.604 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:55:07.061 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:55:07.070 INFO:teuthology.orchestra.run.trial183.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-20T11:55:07.078 INFO:teuthology.orchestra.run.trial183.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-20T11:55:07.083 INFO:teuthology.orchestra.run.trial183.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-20T11:55:07.088 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-20T11:55:07.091 INFO:teuthology.orchestra.run.trial183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-20T11:55:07.095 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-20T11:55:07.096 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-20T11:55:07.113 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-20T11:55:07.116 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-20T11:55:07.125 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-20T11:55:07.134 INFO:teuthology.orchestra.run.trial183.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-20T11:55:07.135 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 11/12 2026-02-20T11:55:08.206 INFO:teuthology.orchestra.run.trial153.stdout:(96/96): ceph-osd-crimson-20.3.0-5357.g0c3148ca 2.2 MB/s | 88 MB 00:40 2026-02-20T11:55:08.208 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:08.209 INFO:teuthology.orchestra.run.trial153.stdout:Total 5.5 MB/s | 226 MB 00:41 2026-02-20T11:55:08.606 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:55:08.644 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:55:08.644 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:55:09.448 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:55:09.448 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:55:10.287 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:55:10.289 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-20T11:55:10.292 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-20T11:55:10.314 INFO:teuthology.orchestra.run.trial153.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-20T11:55:10.347 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-20T11:55:10.354 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-20T11:55:10.356 INFO:teuthology.orchestra.run.trial153.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-20T11:55:10.361 INFO:teuthology.orchestra.run.trial153.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-20T11:55:10.384 INFO:teuthology.orchestra.run.trial153.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-20T11:55:10.397 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-20T11:55:10.400 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-20T11:55:10.406 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-20T11:55:10.408 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-20T11:55:10.419 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-20T11:55:10.425 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-20T11:55:10.449 INFO:teuthology.orchestra.run.trial153.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-20T11:55:10.453 INFO:teuthology.orchestra.run.trial153.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-20T11:55:10.502 INFO:teuthology.orchestra.run.trial153.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-20T11:55:10.515 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-20T11:55:10.518 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-20T11:55:10.522 INFO:teuthology.orchestra.run.trial153.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-20T11:55:10.527 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-20T11:55:10.532 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-20T11:55:10.535 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-20T11:55:10.550 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-20T11:55:10.554 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-20T11:55:10.559 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-20T11:55:10.571 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-20T11:55:10.580 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-20T11:55:10.586 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-20T11:55:10.605 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-20T11:55:10.616 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-20T11:55:10.632 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-20T11:55:10.636 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-20T11:55:10.644 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-20T11:55:10.684 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-20T11:55:11.012 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-20T11:55:11.022 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-20T11:55:11.026 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-20T11:55:11.031 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-20T11:55:11.033 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-20T11:55:11.061 INFO:teuthology.orchestra.run.trial153.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-20T11:55:11.105 INFO:teuthology.orchestra.run.trial153.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-20T11:55:11.113 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-20T11:55:11.119 INFO:teuthology.orchestra.run.trial153.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-20T11:55:11.122 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-20T11:55:11.128 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-20T11:55:11.132 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-20T11:55:11.139 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-20T11:55:11.142 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-20T11:55:11.170 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-20T11:55:11.180 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-20T11:55:11.186 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-20T11:55:11.192 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-20T11:55:11.227 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-20T11:55:11.454 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-20T11:55:11.477 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-20T11:55:11.481 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-20T11:55:11.540 INFO:teuthology.orchestra.run.trial153.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-20T11:55:11.542 INFO:teuthology.orchestra.run.trial153.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-20T11:55:11.563 INFO:teuthology.orchestra.run.trial153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-20T11:55:11.888 INFO:teuthology.orchestra.run.trial153.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-20T11:55:11.961 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-20T11:55:12.637 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-20T11:55:12.654 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-20T11:55:12.664 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-20T11:55:12.671 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-20T11:55:12.684 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-20T11:55:12.700 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-20T11:55:12.776 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-20T11:55:12.785 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-20T11:55:12.808 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-20T11:55:12.838 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-20T11:55:12.889 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-20T11:55:12.895 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-20T11:55:12.898 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-20T11:55:12.899 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-20T11:55:12.918 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:55:12.918 INFO:teuthology.orchestra.run.trial153.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-20T11:55:12.918 INFO:teuthology.orchestra.run.trial153.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-20T11:55:12.918 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:12.927 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:55:12.953 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:55:12.953 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-20T11:55:12.953 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:12.987 INFO:teuthology.orchestra.run.trial153.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-20T11:55:13.578 INFO:teuthology.orchestra.run.trial153.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-20T11:55:14.315 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 11/12 2026-02-20T11:55:14.333 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/12 2026-02-20T11:55:15.099 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 12/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-20T11:55:15.100 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-20T11:55:15.153 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-20T11:55:15.153 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:15.153 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:55:15.153 INFO:teuthology.orchestra.run.trial183.stdout: ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:15.153 INFO:teuthology.orchestra.run.trial183.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-20T11:55:15.153 INFO:teuthology.orchestra.run.trial183.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout: libcephsqlite-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:15.154 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:15.195 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph 2026-02-20T11:55:15.501 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:01:27 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:15.587 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:15.589 INFO:teuthology.orchestra.run.trial183.stdout:======================================================================================= 2026-02-20T11:55:15.590 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repository Size 2026-02-20T11:55:15.590 INFO:teuthology.orchestra.run.trial183.stdout:======================================================================================= 2026-02-20T11:55:15.590 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:55:15.590 INFO:teuthology.orchestra.run.trial183.stdout: ceph x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 6.5 k 2026-02-20T11:55:15.590 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-20T11:55:15.590 INFO:teuthology.orchestra.run.trial183.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-20T11:55:15.590 INFO:teuthology.orchestra.run.trial183.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 57 k 2026-02-20T11:55:15.590 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mds x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.0 M 2026-02-20T11:55:15.590 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 937 k 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 297 k 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mon x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 4.9 M 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 163 k 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-classic x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 15 M 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-crimson x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 88 M 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 17 k 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: cephadm noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 1.0 M 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-20T11:55:15.591 INFO:teuthology.orchestra.run.trial183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-20T11:55:15.592 INFO:teuthology.orchestra.run.trial183.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-20T11:55:15.593 INFO:teuthology.orchestra.run.trial183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-20T11:55:15.593 INFO:teuthology.orchestra.run.trial183.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-20T11:55:15.593 INFO:teuthology.orchestra.run.trial183.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-20T11:55:15.593 INFO:teuthology.orchestra.run.trial183.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-20T11:55:15.593 INFO:teuthology.orchestra.run.trial183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-20T11:55:15.593 INFO:teuthology.orchestra.run.trial183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-20T11:55:15.593 INFO:teuthology.orchestra.run.trial183.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-20T11:55:15.593 INFO:teuthology.orchestra.run.trial183.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-20T11:55:15.593 INFO:teuthology.orchestra.run.trial183.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-20T11:55:15.593 INFO:teuthology.orchestra.run.trial183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-20T11:55:15.594 INFO:teuthology.orchestra.run.trial183.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-20T11:55:15.594 INFO:teuthology.orchestra.run.trial183.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-20T11:55:15.594 INFO:teuthology.orchestra.run.trial183.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-20T11:55:15.594 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-20T11:55:15.594 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-20T11:55:15.594 INFO:teuthology.orchestra.run.trial183.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-20T11:55:15.594 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-20T11:55:15.594 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-20T11:55:15.594 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-20T11:55:15.594 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-20T11:55:15.595 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-20T11:55:15.596 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-20T11:55:15.597 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout:Installing weak dependencies: 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 184 k 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 51 M 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 7.4 M 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 22 k 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-rook noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 50 k 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: ceph-volume noarch 2:20.3.0-5357.g0c3148ca.el9 ceph-noarch 301 k 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-20T11:55:15.598 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:15.599 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:55:15.599 INFO:teuthology.orchestra.run.trial183.stdout:======================================================================================= 2026-02-20T11:55:15.599 INFO:teuthology.orchestra.run.trial183.stdout:Install 96 Packages 2026-02-20T11:55:15.599 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:15.600 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 226 M 2026-02-20T11:55:15.600 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 1.5 G 2026-02-20T11:55:15.600 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:55:16.439 INFO:teuthology.orchestra.run.trial183.stdout:(1/96): ceph-20.3.0-5357.g0c3148ca.el9.x86_64.r 25 kB/s | 6.5 kB 00:00 2026-02-20T11:55:17.036 INFO:teuthology.orchestra.run.trial183.stdout:(2/96): ceph-mgr-20.3.0-5357.g0c3148ca.el9.x86_ 1.1 MB/s | 937 kB 00:00 2026-02-20T11:55:17.161 INFO:teuthology.orchestra.run.trial183.stdout:(3/96): ceph-osd-20.3.0-5357.g0c3148ca.el9.x86_ 1.3 MB/s | 163 kB 00:00 2026-02-20T11:55:17.218 INFO:teuthology.orchestra.run.trial183.stdout:(4/96): ceph-mds-20.3.0-5357.g0c3148ca.el9.x86_ 1.9 MB/s | 2.0 MB 00:01 2026-02-20T11:55:18.987 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 80/96 2026-02-20T11:55:19.052 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 81/96 2026-02-20T11:55:19.054 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 82/96 2026-02-20T11:55:19.068 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 82/96 2026-02-20T11:55:19.068 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:19.068 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-20T11:55:19.068 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:19.079 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 83/96 2026-02-20T11:55:19.102 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 83/96 2026-02-20T11:55:19.102 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:19.102 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-20T11:55:19.102 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-20T11:55:19.102 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-20T11:55:19.102 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:19.123 INFO:teuthology.orchestra.run.trial183.stdout:(5/96): ceph-mon-20.3.0-5357.g0c3148ca.el9.x86_ 1.8 MB/s | 4.9 MB 00:02 2026-02-20T11:55:19.163 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 84/96 2026-02-20T11:55:19.165 INFO:teuthology.orchestra.run.trial153.stdout: Installing : cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 84/96 2026-02-20T11:55:19.174 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9 85/96 2026-02-20T11:55:19.187 INFO:teuthology.orchestra.run.trial183.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5357.g0c 901 kB/s | 57 kB 00:00 2026-02-20T11:55:19.198 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el 86/96 2026-02-20T11:55:19.199 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 87/96 2026-02-20T11:55:19.307 INFO:teuthology.orchestra.run.trial183.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5357.g0c3148ca. 1.5 MB/s | 184 kB 00:00 2026-02-20T11:55:22.101 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 87/96 2026-02-20T11:55:22.108 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 88/96 2026-02-20T11:55:22.303 INFO:teuthology.orchestra.run.trial183.stdout:(8/96): ceph-osd-classic-20.3.0-5357.g0c3148ca. 2.9 MB/s | 15 MB 00:05 2026-02-20T11:55:22.566 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 88/96 2026-02-20T11:55:22.567 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 89/96 2026-02-20T11:55:22.579 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 89/96 2026-02-20T11:55:22.580 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 90/96 2026-02-20T11:55:22.643 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 90/96 2026-02-20T11:55:22.683 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9. 91/96 2026-02-20T11:55:22.684 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 92/96 2026-02-20T11:55:22.706 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 92/96 2026-02-20T11:55:22.706 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:22.706 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-20T11:55:22.706 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-20T11:55:22.706 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-20T11:55:22.706 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:22.718 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 93/96 2026-02-20T11:55:22.933 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 93/96 2026-02-20T11:55:22.934 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 94/96 2026-02-20T11:55:22.955 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 94/96 2026-02-20T11:55:22.955 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:22.956 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-20T11:55:22.956 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-20T11:55:22.956 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-20T11:55:22.956 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:23.039 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 95/96 2026-02-20T11:55:23.058 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 95/96 2026-02-20T11:55:23.058 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:23.059 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-20T11:55:23.059 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-20T11:55:23.059 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-20T11:55:23.059 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:23.060 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 96/96 2026-02-20T11:55:23.066 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 96/96 2026-02-20T11:55:23.090 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 96/96 2026-02-20T11:55:24.499 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 96/96 2026-02-20T11:55:24.499 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/96 2026-02-20T11:55:24.499 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/96 2026-02-20T11:55:24.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/96 2026-02-20T11:55:24.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 4/96 2026-02-20T11:55:24.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 5/96 2026-02-20T11:55:24.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 6/96 2026-02-20T11:55:24.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 7/96 2026-02-20T11:55:24.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el 8/96 2026-02-20T11:55:24.500 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 9/96 2026-02-20T11:55:24.501 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 10/96 2026-02-20T11:55:24.501 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 11/96 2026-02-20T11:55:24.501 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 12/96 2026-02-20T11:55:24.501 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9. 13/96 2026-02-20T11:55:24.501 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 14/96 2026-02-20T11:55:24.501 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9 15/96 2026-02-20T11:55:24.501 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 16/96 2026-02-20T11:55:24.501 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 17/96 2026-02-20T11:55:24.501 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-20T11:55:24.502 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-20T11:55:24.502 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-20T11:55:24.502 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-20T11:55:24.502 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-20T11:55:24.502 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-20T11:55:24.502 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-20T11:55:24.502 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-20T11:55:24.502 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-20T11:55:24.503 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-20T11:55:24.503 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-20T11:55:24.503 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-20T11:55:24.503 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-20T11:55:24.503 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-20T11:55:24.503 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-20T11:55:24.503 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-20T11:55:24.503 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-20T11:55:24.503 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-20T11:55:24.503 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-20T11:55:24.504 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-20T11:55:24.505 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-20T11:55:24.506 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-20T11:55:24.507 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-20T11:55:24.508 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-20T11:55:24.508 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-20T11:55:24.508 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-20T11:55:24.508 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-20T11:55:24.508 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-20T11:55:24.508 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-20T11:55:24.508 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-20T11:55:24.555 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: lua-5.4.4-4.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-20T11:55:24.556 INFO:teuthology.orchestra.run.trial153.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-ply-3.11-14.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-20T11:55:24.557 INFO:teuthology.orchestra.run.trial153.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: unzip-6.0-59.el9.x86_64 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: zip-3.0-35.el9.x86_64 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:24.558 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:24.625 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-base 2026-02-20T11:55:24.629 INFO:teuthology.orchestra.run.trial183.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 3.2 MB/s | 7.4 MB 00:02 2026-02-20T11:55:24.695 INFO:teuthology.orchestra.run.trial183.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5357.g0c3148 337 kB/s | 22 kB 00:00 2026-02-20T11:55:24.832 INFO:teuthology.orchestra.run.trial183.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5357.g0c3 2.1 MB/s | 297 kB 00:00 2026-02-20T11:55:24.900 INFO:teuthology.orchestra.run.trial183.stdout:(12/96): ceph-mgr-rook-20.3.0-5357.g0c3148ca.el 740 kB/s | 50 kB 00:00 2026-02-20T11:55:24.933 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:32 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:24.967 INFO:teuthology.orchestra.run.trial183.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5357.g0c 261 kB/s | 17 kB 00:00 2026-02-20T11:55:24.998 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:25.016 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:25.017 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:25.017 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:25.038 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install cephadm 2026-02-20T11:55:25.105 INFO:teuthology.orchestra.run.trial183.stdout:(14/96): ceph-volume-20.3.0-5357.g0c3148ca.el9. 2.1 MB/s | 301 kB 00:00 2026-02-20T11:55:25.345 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:33 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:25.411 INFO:teuthology.orchestra.run.trial153.stdout:Package cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:25.430 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:25.430 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:25.430 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:25.452 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-immutable-object-cache 2026-02-20T11:55:25.520 INFO:teuthology.orchestra.run.trial183.stdout:(15/96): cephadm-20.3.0-5357.g0c3148ca.el9.noar 2.4 MB/s | 1.0 MB 00:00 2026-02-20T11:55:25.757 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:33 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:25.842 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:25.842 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:25.842 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-20T11:55:25.842 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:25.842 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:55:25.843 INFO:teuthology.orchestra.run.trial153.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 149 k 2026-02-20T11:55:25.843 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:25.843 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:55:25.843 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:25.843 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-20T11:55:25.843 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:25.843 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 149 k 2026-02-20T11:55:25.843 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 455 k 2026-02-20T11:55:25.843 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:55:26.335 INFO:teuthology.orchestra.run.trial153.stdout:ceph-immutable-object-cache-20.3.0-5357.g0c3148 304 kB/s | 149 kB 00:00 2026-02-20T11:55:26.337 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:26.337 INFO:teuthology.orchestra.run.trial153.stdout:Total 302 kB/s | 149 kB 00:00 2026-02-20T11:55:26.337 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:55:26.342 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:55:26.342 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:55:26.373 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:55:26.374 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:55:26.429 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:55:26.430 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:55:26.454 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:55:26.454 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:26.454 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-20T11:55:26.455 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:26.570 INFO:teuthology.orchestra.run.trial183.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 334 kB/s | 351 kB 00:01 2026-02-20T11:55:26.740 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:55:26.740 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:26.740 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:55:26.740 INFO:teuthology.orchestra.run.trial153.stdout: ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:26.741 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:26.741 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:26.745 INFO:teuthology.orchestra.run.trial183.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 231 kB/s | 40 kB 00:00 2026-02-20T11:55:26.768 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr 2026-02-20T11:55:27.006 INFO:teuthology.orchestra.run.trial183.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 276 kB/s | 72 kB 00:00 2026-02-20T11:55:27.079 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:35 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:27.144 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:27.163 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:27.163 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:27.164 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:27.184 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-dashboard 2026-02-20T11:55:27.487 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:35 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:27.553 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:27.571 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:27.572 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:27.572 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:27.593 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-20T11:55:27.904 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:35 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:27.970 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:27.988 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:27.989 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:27.989 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:28.010 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-rook 2026-02-20T11:55:28.135 INFO:teuthology.orchestra.run.trial183.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 703 kB/s | 794 kB 00:01 2026-02-20T11:55:28.317 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:36 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:28.381 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:28.400 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:28.400 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:28.400 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:28.422 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-cephadm 2026-02-20T11:55:28.490 INFO:teuthology.orchestra.run.trial183.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 520 kB/s | 184 kB 00:00 2026-02-20T11:55:28.732 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:36 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:28.797 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:28.815 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:28.816 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:28.816 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:28.837 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-osd 2026-02-20T11:55:28.926 INFO:teuthology.orchestra.run.trial183.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 581 kB/s | 253 kB 00:00 2026-02-20T11:55:29.145 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:37 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:29.211 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:29.229 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:29.230 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:29.230 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:29.251 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-osd-classic 2026-02-20T11:55:29.564 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:37 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:29.627 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:29.646 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:29.647 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:29.647 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:29.668 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-fuse 2026-02-20T11:55:29.976 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:37 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:30.061 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:30.061 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout: ceph-fuse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 914 k 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 914 k 2026-02-20T11:55:30.062 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 2.9 M 2026-02-20T11:55:30.063 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:55:30.663 INFO:teuthology.orchestra.run.trial183.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 734 kB/s | 1.2 MB 00:01 2026-02-20T11:55:30.922 INFO:teuthology.orchestra.run.trial153.stdout:ceph-fuse-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 1.0 MB/s | 914 kB 00:00 2026-02-20T11:55:30.922 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:30.923 INFO:teuthology.orchestra.run.trial153.stdout:Total 1.0 MB/s | 914 kB 00:00 2026-02-20T11:55:30.923 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:55:30.927 INFO:teuthology.orchestra.run.trial183.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 404 kB/s | 106 kB 00:00 2026-02-20T11:55:30.928 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:55:30.928 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:55:30.963 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:55:30.963 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:55:31.056 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:55:31.074 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:31.189 INFO:teuthology.orchestra.run.trial183.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 515 kB/s | 135 kB 00:00 2026-02-20T11:55:31.299 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:31.356 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:31.356 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:31.356 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:55:31.356 INFO:teuthology.orchestra.run.trial153.stdout: ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:31.356 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:31.356 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:31.383 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-volume 2026-02-20T11:55:31.538 INFO:teuthology.orchestra.run.trial183.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 432 kB/s | 150 kB 00:00 2026-02-20T11:55:31.692 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:39 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:31.758 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:31.776 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:31.777 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:31.777 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:31.798 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install librados-devel 2026-02-20T11:55:31.801 INFO:teuthology.orchestra.run.trial183.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 482 kB/s | 126 kB 00:00 2026-02-20T11:55:32.110 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:40 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:32.193 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:32.193 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:32.193 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-20T11:55:32.193 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:32.194 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:55:32.194 INFO:teuthology.orchestra.run.trial153.stdout: librados-devel x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 124 k 2026-02-20T11:55:32.194 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:32.194 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:55:32.194 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:32.194 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-20T11:55:32.194 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:32.194 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 124 k 2026-02-20T11:55:32.194 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 434 k 2026-02-20T11:55:32.194 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:55:32.236 INFO:teuthology.orchestra.run.trial183.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 500 kB/s | 218 kB 00:00 2026-02-20T11:55:32.589 INFO:teuthology.orchestra.run.trial183.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 516 kB/s | 182 kB 00:00 2026-02-20T11:55:32.676 INFO:teuthology.orchestra.run.trial153.stdout:librados-devel-20.3.0-5357.g0c3148ca.el9.x86_64 257 kB/s | 124 kB 00:00 2026-02-20T11:55:32.676 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:32.676 INFO:teuthology.orchestra.run.trial153.stdout:Total 256 kB/s | 124 kB 00:00 2026-02-20T11:55:32.677 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:55:32.679 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:55:32.679 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:55:32.714 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:55:32.714 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:55:32.770 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:55:32.787 INFO:teuthology.orchestra.run.trial153.stdout: Installing : librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:32.871 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:32.928 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:32.928 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:32.929 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:55:32.929 INFO:teuthology.orchestra.run.trial153.stdout: librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:32.929 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:32.929 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:32.957 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install libcephfs2 2026-02-20T11:55:33.024 INFO:teuthology.orchestra.run.trial183.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 611 kB/s | 266 kB 00:00 2026-02-20T11:55:33.270 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:41 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:33.336 INFO:teuthology.orchestra.run.trial153.stdout:Package libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:33.355 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:33.355 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:33.355 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:33.376 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install libcephfs-devel 2026-02-20T11:55:33.395 INFO:teuthology.orchestra.run.trial183.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 80 kB/s | 30 kB 00:00 2026-02-20T11:55:33.684 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:41 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:33.767 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:33.767 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-devel x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 37 k 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 37 k 2026-02-20T11:55:33.768 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 171 k 2026-02-20T11:55:33.769 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:55:34.015 INFO:teuthology.orchestra.run.trial183.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 4.8 MB/s | 3.0 MB 00:00 2026-02-20T11:55:34.088 INFO:teuthology.orchestra.run.trial153.stdout:libcephfs-devel-20.3.0-5357.g0c3148ca.el9.x86_6 116 kB/s | 37 kB 00:00 2026-02-20T11:55:34.089 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:34.089 INFO:teuthology.orchestra.run.trial153.stdout:Total 116 kB/s | 37 kB 00:00 2026-02-20T11:55:34.089 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:55:34.092 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:55:34.092 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:55:34.094 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:55:34.094 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:55:34.096 INFO:teuthology.orchestra.run.trial183.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 182 kB/s | 15 kB 00:00 2026-02-20T11:55:34.116 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:55:34.132 INFO:teuthology.orchestra.run.trial153.stdout: Installing : libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:34.181 INFO:teuthology.orchestra.run.trial183.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 2.8 MB/s | 246 kB 00:00 2026-02-20T11:55:34.188 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:34.245 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:34.245 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:34.245 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:55:34.245 INFO:teuthology.orchestra.run.trial153.stdout: libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:34.245 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:34.246 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:34.265 INFO:teuthology.orchestra.run.trial183.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2026-02-20T11:55:34.271 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install librados2 2026-02-20T11:55:34.347 INFO:teuthology.orchestra.run.trial183.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 516 kB/s | 42 kB 00:00 2026-02-20T11:55:34.562 INFO:teuthology.orchestra.run.trial183.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 25 MB/s | 5.3 MB 00:00 2026-02-20T11:55:34.580 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:42 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:34.645 INFO:teuthology.orchestra.run.trial153.stdout:Package librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:34.664 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:34.664 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:34.664 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:34.685 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install librbd1 2026-02-20T11:55:34.757 INFO:teuthology.orchestra.run.trial183.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 6.0 MB 00:00 2026-02-20T11:55:34.841 INFO:teuthology.orchestra.run.trial183.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 2.8 MB/s | 244 kB 00:00 2026-02-20T11:55:34.926 INFO:teuthology.orchestra.run.trial183.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 2.9 MB/s | 249 kB 00:00 2026-02-20T11:55:34.992 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:42 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:35.008 INFO:teuthology.orchestra.run.trial183.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 577 kB/s | 48 kB 00:00 2026-02-20T11:55:35.058 INFO:teuthology.orchestra.run.trial153.stdout:Package librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:35.076 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:35.077 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:35.077 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:35.091 INFO:teuthology.orchestra.run.trial183.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 2.1 MB/s | 177 kB 00:00 2026-02-20T11:55:35.098 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-rados 2026-02-20T11:55:35.183 INFO:teuthology.orchestra.run.trial183.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 13 MB/s | 1.2 MB 00:00 2026-02-20T11:55:35.265 INFO:teuthology.orchestra.run.trial183.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 422 kB/s | 35 kB 00:00 2026-02-20T11:55:35.407 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:43 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:35.460 INFO:teuthology.orchestra.run.trial183.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 32 MB/s | 6.1 MB 00:00 2026-02-20T11:55:35.473 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:35.491 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:35.492 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:35.492 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:35.513 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-rgw 2026-02-20T11:55:35.545 INFO:teuthology.orchestra.run.trial183.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 5.1 MB/s | 442 kB 00:00 2026-02-20T11:55:35.628 INFO:teuthology.orchestra.run.trial183.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 932 kB/s | 77 kB 00:00 2026-02-20T11:55:35.712 INFO:teuthology.orchestra.run.trial183.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 3.1 MB/s | 267 kB 00:00 2026-02-20T11:55:35.827 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:43 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:35.847 INFO:teuthology.orchestra.run.trial183.stdout:(48/96): ceph-osd-crimson-20.3.0-5357.g0c3148ca 4.7 MB/s | 88 MB 00:18 2026-02-20T11:55:35.848 INFO:teuthology.orchestra.run.trial183.stdout:(49/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2026-02-20T11:55:35.893 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:35.911 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:35.912 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:35.912 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:35.931 INFO:teuthology.orchestra.run.trial183.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 649 kB/s | 54 kB 00:00 2026-02-20T11:55:35.934 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-cephfs 2026-02-20T11:55:36.243 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:44 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:36.306 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:36.325 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:36.325 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:36.326 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:36.348 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-rbd 2026-02-20T11:55:36.379 INFO:teuthology.orchestra.run.trial183.stdout:(51/96): python3-pyasn1-modules-0.4.8-6.el9.noa 524 kB/s | 279 kB 00:00 2026-02-20T11:55:36.468 INFO:teuthology.orchestra.run.trial183.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 467 kB/s | 42 kB 00:00 2026-02-20T11:55:36.572 INFO:teuthology.orchestra.run.trial183.stdout:(53/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 30 MB/s | 19 MB 00:00 2026-02-20T11:55:36.588 INFO:teuthology.orchestra.run.trial183.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 1.5 MB/s | 189 kB 00:00 2026-02-20T11:55:36.631 INFO:teuthology.orchestra.run.trial183.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 523 kB/s | 22 kB 00:00 2026-02-20T11:55:36.654 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:44 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:36.655 INFO:teuthology.orchestra.run.trial183.stdout:(56/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 1.1 MB/s | 90 kB 00:00 2026-02-20T11:55:36.717 INFO:teuthology.orchestra.run.trial183.stdout:(57/96): protobuf-compiler-3.14.0-17.el9.x86_64 9.8 MB/s | 862 kB 00:00 2026-02-20T11:55:36.720 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:36.738 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:36.739 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:36.739 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:36.760 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install rbd-fuse 2026-02-20T11:55:36.820 INFO:teuthology.orchestra.run.trial183.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 188 kB/s | 19 kB 00:00 2026-02-20T11:55:36.904 INFO:teuthology.orchestra.run.trial183.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2026-02-20T11:55:36.930 INFO:teuthology.orchestra.run.trial183.stdout:(60/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.0 MB/s | 551 kB 00:00 2026-02-20T11:55:36.960 INFO:teuthology.orchestra.run.trial183.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 1.0 MB/s | 29 kB 00:00 2026-02-20T11:55:36.989 INFO:teuthology.orchestra.run.trial183.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 2.0 MB/s | 60 kB 00:00 2026-02-20T11:55:36.991 INFO:teuthology.orchestra.run.trial183.stdout:(63/96): python3-asyncssh-2.13.2-5.el9.noarch.r 6.1 MB/s | 548 kB 00:00 2026-02-20T11:55:37.018 INFO:teuthology.orchestra.run.trial183.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.5 MB/s | 43 kB 00:00 2026-02-20T11:55:37.020 INFO:teuthology.orchestra.run.trial183.stdout:(65/96): python3-cachetools-4.2.4-1.el9.noarch. 1.1 MB/s | 32 kB 00:00 2026-02-20T11:55:37.047 INFO:teuthology.orchestra.run.trial183.stdout:(66/96): python3-certifi-2023.05.07-4.el9.noarc 492 kB/s | 14 kB 00:00 2026-02-20T11:55:37.052 INFO:teuthology.orchestra.run.trial183.stdout:(67/96): python3-cheroot-10.0.1-4.el9.noarch.rp 5.5 MB/s | 173 kB 00:00 2026-02-20T11:55:37.067 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:45 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:37.079 INFO:teuthology.orchestra.run.trial183.stdout:(68/96): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2026-02-20T11:55:37.111 INFO:teuthology.orchestra.run.trial183.stdout:(69/96): python3-google-auth-2.45.0-1.el9.noarc 4.2 MB/s | 254 kB 00:00 2026-02-20T11:55:37.142 INFO:teuthology.orchestra.run.trial183.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 4.6 MB/s | 144 kB 00:00 2026-02-20T11:55:37.151 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:37.151 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:37.152 INFO:teuthology.orchestra.run.trial153.stdout: Package Architecture Version Repository Size 2026-02-20T11:55:37.152 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:37.152 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:55:37.152 INFO:teuthology.orchestra.run.trial153.stdout: rbd-fuse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 86 k 2026-02-20T11:55:37.152 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:37.152 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:55:37.152 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:37.152 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-20T11:55:37.152 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:37.152 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 86 k 2026-02-20T11:55:37.153 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 236 k 2026-02-20T11:55:37.153 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:55:37.173 INFO:teuthology.orchestra.run.trial183.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 4.3 MB/s | 139 kB 00:00 2026-02-20T11:55:37.198 INFO:teuthology.orchestra.run.trial183.stdout:(72/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 17 MB/s | 2.0 MB 00:00 2026-02-20T11:55:37.203 INFO:teuthology.orchestra.run.trial183.stdout:(73/96): python3-isodate-0.6.1-3.el9.noarch.rpm 1.8 MB/s | 56 kB 00:00 2026-02-20T11:55:37.227 INFO:teuthology.orchestra.run.trial183.stdout:(74/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 369 kB/s | 11 kB 00:00 2026-02-20T11:55:37.232 INFO:teuthology.orchestra.run.trial183.stdout:(75/96): python3-jaraco-classes-3.2.1-5.el9.noa 608 kB/s | 18 kB 00:00 2026-02-20T11:55:37.256 INFO:teuthology.orchestra.run.trial183.stdout:(76/96): python3-jaraco-collections-3.0.0-8.el9 805 kB/s | 23 kB 00:00 2026-02-20T11:55:37.261 INFO:teuthology.orchestra.run.trial183.stdout:(77/96): python3-jaraco-context-6.0.1-3.el9.noa 682 kB/s | 20 kB 00:00 2026-02-20T11:55:37.285 INFO:teuthology.orchestra.run.trial183.stdout:(78/96): python3-jaraco-functools-3.5.0-2.el9.n 675 kB/s | 19 kB 00:00 2026-02-20T11:55:37.290 INFO:teuthology.orchestra.run.trial183.stdout:(79/96): python3-jaraco-text-4.0.0-2.el9.noarch 911 kB/s | 26 kB 00:00 2026-02-20T11:55:37.320 INFO:teuthology.orchestra.run.trial183.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 2.6 MB/s | 79 kB 00:00 2026-02-20T11:55:37.345 INFO:teuthology.orchestra.run.trial183.stdout:(81/96): python3-kubernetes-26.1.0-3.el9.noarch 17 MB/s | 1.0 MB 00:00 2026-02-20T11:55:37.350 INFO:teuthology.orchestra.run.trial183.stdout:(82/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 2.8 MB/s | 86 kB 00:00 2026-02-20T11:55:37.375 INFO:teuthology.orchestra.run.trial183.stdout:(83/96): python3-natsort-7.1.1-5.el9.noarch.rpm 1.9 MB/s | 58 kB 00:00 2026-02-20T11:55:37.379 INFO:teuthology.orchestra.run.trial183.stdout:(84/96): python3-portend-3.1.0-2.el9.noarch.rpm 567 kB/s | 16 kB 00:00 2026-02-20T11:55:37.405 INFO:teuthology.orchestra.run.trial183.stdout:(85/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.0 MB/s | 90 kB 00:00 2026-02-20T11:55:37.408 INFO:teuthology.orchestra.run.trial183.stdout:(86/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.0 MB/s | 31 kB 00:00 2026-02-20T11:55:37.436 INFO:teuthology.orchestra.run.trial183.stdout:(87/96): python3-routes-2.5.1-5.el9.noarch.rpm 6.0 MB/s | 188 kB 00:00 2026-02-20T11:55:37.438 INFO:teuthology.orchestra.run.trial183.stdout:(88/96): python3-rsa-4.9-2.el9.noarch.rpm 2.0 MB/s | 59 kB 00:00 2026-02-20T11:55:37.466 INFO:teuthology.orchestra.run.trial183.stdout:(89/96): python3-saml-1.16.0-1.el9.noarch.rpm 4.0 MB/s | 125 kB 00:00 2026-02-20T11:55:37.467 INFO:teuthology.orchestra.run.trial183.stdout:(90/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.2 MB/s | 36 kB 00:00 2026-02-20T11:55:37.495 INFO:teuthology.orchestra.run.trial183.stdout:(91/96): python3-typing-extensions-4.15.0-1.el9 2.9 MB/s | 86 kB 00:00 2026-02-20T11:55:37.497 INFO:teuthology.orchestra.run.trial183.stdout:(92/96): python3-websocket-client-1.2.3-2.el9.n 2.9 MB/s | 90 kB 00:00 2026-02-20T11:55:37.525 INFO:teuthology.orchestra.run.trial183.stdout:(93/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 1.6 MB/s | 48 kB 00:00 2026-02-20T11:55:37.526 INFO:teuthology.orchestra.run.trial183.stdout:(94/96): python3-xmltodict-0.12.0-15.el9.noarch 772 kB/s | 22 kB 00:00 2026-02-20T11:55:37.534 INFO:teuthology.orchestra.run.trial153.stdout:rbd-fuse-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 226 kB/s | 86 kB 00:00 2026-02-20T11:55:37.535 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:37.535 INFO:teuthology.orchestra.run.trial153.stdout:Total 226 kB/s | 86 kB 00:00 2026-02-20T11:55:37.535 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:55:37.539 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:55:37.539 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:55:37.554 INFO:teuthology.orchestra.run.trial183.stdout:(95/96): python3-zc-lockfile-2.0-10.el9.noarch. 685 kB/s | 20 kB 00:00 2026-02-20T11:55:37.570 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:55:37.570 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:55:37.621 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:55:37.638 INFO:teuthology.orchestra.run.trial153.stdout: Installing : rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:37.727 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:37.786 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:37.786 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:37.786 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:55:37.786 INFO:teuthology.orchestra.run.trial153.stdout: rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:37.786 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:37.786 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:37.807 INFO:teuthology.orchestra.run.trial183.stdout:(96/96): ceph-mgr-dashboard-20.3.0-5357.g0c3148 2.7 MB/s | 51 MB 00:18 2026-02-20T11:55:37.809 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:37.809 INFO:teuthology.orchestra.run.trial183.stdout:Total 10 MB/s | 226 MB 00:22 2026-02-20T11:55:37.813 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install rbd-mirror 2026-02-20T11:55:38.129 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:46 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:38.214 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:55:38.215 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:38.215 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout: rbd-mirror x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.6 M 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 2.6 M 2026-02-20T11:55:38.216 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 11 M 2026-02-20T11:55:38.217 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:55:38.253 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:55:38.253 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:55:38.720 INFO:teuthology.orchestra.run.trial153.stdout:rbd-mirror-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 5.1 MB/s | 2.6 MB 00:00 2026-02-20T11:55:38.720 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:38.721 INFO:teuthology.orchestra.run.trial153.stdout:Total 5.1 MB/s | 2.6 MB 00:00 2026-02-20T11:55:38.721 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:55:38.726 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:55:38.726 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:55:38.766 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:55:38.766 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:55:38.934 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:55:38.936 INFO:teuthology.orchestra.run.trial153.stdout: Installing : rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:38.958 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:38.959 INFO:teuthology.orchestra.run.trial153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:38.959 INFO:teuthology.orchestra.run.trial153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-20T11:55:38.959 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T11:55:38.959 INFO:teuthology.orchestra.run.trial153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T11:55:38.959 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:39.051 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:55:39.052 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:55:39.244 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:39.245 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:39.245 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:55:39.245 INFO:teuthology.orchestra.run.trial153.stdout: rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:39.245 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:39.245 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:39.273 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install rbd-nbd 2026-02-20T11:55:39.586 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:47 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:39.669 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:39.670 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:39.670 INFO:teuthology.orchestra.run.trial153.stdout: Package Architecture Version Repository Size 2026-02-20T11:55:39.670 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:39.670 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:55:39.670 INFO:teuthology.orchestra.run.trial153.stdout: rbd-nbd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 167 k 2026-02-20T11:55:39.670 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:39.670 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:55:39.670 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:39.670 INFO:teuthology.orchestra.run.trial153.stdout:Install 1 Package 2026-02-20T11:55:39.670 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:39.671 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 167 k 2026-02-20T11:55:39.671 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 496 k 2026-02-20T11:55:39.671 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:55:39.891 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:55:39.893 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-20T11:55:39.896 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-20T11:55:39.918 INFO:teuthology.orchestra.run.trial183.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-20T11:55:39.952 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-20T11:55:39.958 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-20T11:55:39.961 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-20T11:55:39.965 INFO:teuthology.orchestra.run.trial183.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-20T11:55:39.970 INFO:teuthology.orchestra.run.trial153.stdout:rbd-nbd-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 556 kB/s | 167 kB 00:00 2026-02-20T11:55:39.970 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:39.971 INFO:teuthology.orchestra.run.trial153.stdout:Total 554 kB/s | 167 kB 00:00 2026-02-20T11:55:39.971 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:55:39.975 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:55:39.975 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:55:39.988 INFO:teuthology.orchestra.run.trial183.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-20T11:55:40.001 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-20T11:55:40.004 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-20T11:55:40.008 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:55:40.008 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:55:40.010 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-20T11:55:40.012 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-20T11:55:40.023 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-20T11:55:40.029 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-20T11:55:40.053 INFO:teuthology.orchestra.run.trial183.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-20T11:55:40.056 INFO:teuthology.orchestra.run.trial183.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-20T11:55:40.066 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:55:40.082 INFO:teuthology.orchestra.run.trial153.stdout: Installing : rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:40.107 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-20T11:55:40.119 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-20T11:55:40.122 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-20T11:55:40.126 INFO:teuthology.orchestra.run.trial183.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-20T11:55:40.131 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-20T11:55:40.136 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-20T11:55:40.139 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-20T11:55:40.153 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-20T11:55:40.158 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-20T11:55:40.163 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-20T11:55:40.171 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:40.174 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-20T11:55:40.183 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-20T11:55:40.189 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-20T11:55:40.208 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-20T11:55:40.219 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-20T11:55:40.234 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:55:40.234 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:40.234 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:55:40.234 INFO:teuthology.orchestra.run.trial153.stdout: rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:40.234 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:40.235 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:40.235 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-20T11:55:40.238 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-20T11:55:40.246 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-20T11:55:40.267 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-osd-crimson 2026-02-20T11:55:40.285 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-20T11:55:40.576 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:48 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:40.604 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-20T11:55:40.613 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-20T11:55:40.617 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-20T11:55:40.623 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-20T11:55:40.625 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-20T11:55:40.640 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:40.652 INFO:teuthology.orchestra.run.trial183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-20T11:55:40.659 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:40.660 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:55:40.660 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:55:40.681 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-20T11:55:40.695 INFO:teuthology.orchestra.run.trial183.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-20T11:55:40.703 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-20T11:55:40.709 INFO:teuthology.orchestra.run.trial183.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-20T11:55:40.712 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-20T11:55:40.718 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-20T11:55:40.721 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-20T11:55:40.728 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-20T11:55:40.732 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-20T11:55:40.759 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-20T11:55:40.768 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-20T11:55:40.774 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-20T11:55:40.781 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-20T11:55:40.815 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-20T11:55:40.994 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:01:48 ago on Fri 20 Feb 2026 11:53:52 AM UTC. 2026-02-20T11:55:41.036 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-20T11:55:41.059 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-20T11:55:41.063 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-20T11:55:41.077 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:55:41.078 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:41.078 INFO:teuthology.orchestra.run.trial153.stdout: Package Arch Version Repo Size 2026-02-20T11:55:41.078 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:41.078 INFO:teuthology.orchestra.run.trial153.stdout:Installing: 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 433 M 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout:Installing weak dependencies: 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout: ceph-debugsource x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 12 M 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout:Transaction Summary 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout:================================================================================ 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout:Install 2 Packages 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout:Total download size: 445 M 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout:Installed size: 2.7 G 2026-02-20T11:55:41.079 INFO:teuthology.orchestra.run.trial153.stdout:Downloading Packages: 2026-02-20T11:55:41.120 INFO:teuthology.orchestra.run.trial183.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-20T11:55:41.122 INFO:teuthology.orchestra.run.trial183.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-20T11:55:41.143 INFO:teuthology.orchestra.run.trial183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-20T11:55:41.463 INFO:teuthology.orchestra.run.trial183.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-20T11:55:41.535 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-20T11:55:42.196 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-20T11:55:42.212 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-20T11:55:42.222 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-20T11:55:42.228 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-20T11:55:42.242 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-20T11:55:42.257 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-20T11:55:42.331 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-20T11:55:42.340 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-20T11:55:42.363 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-20T11:55:42.392 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-20T11:55:42.441 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-20T11:55:42.447 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-20T11:55:42.451 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-20T11:55:42.452 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-20T11:55:42.469 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:55:42.470 INFO:teuthology.orchestra.run.trial183.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-20T11:55:42.470 INFO:teuthology.orchestra.run.trial183.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-20T11:55:42.470 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:42.477 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:55:42.502 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-20T11:55:42.502 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-20T11:55:42.503 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:42.539 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-20T11:55:42.682 INFO:teuthology.orchestra.run.trial153.stdout:(1/2): ceph-debugsource-20.3.0-5357.g0c3148ca.e 7.5 MB/s | 12 MB 00:01 2026-02-20T11:55:43.125 INFO:teuthology.orchestra.run.trial183.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-20T11:55:48.524 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 80/96 2026-02-20T11:55:48.587 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 81/96 2026-02-20T11:55:48.590 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 82/96 2026-02-20T11:55:48.604 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 82/96 2026-02-20T11:55:48.604 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:48.604 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-20T11:55:48.604 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:48.616 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 83/96 2026-02-20T11:55:48.641 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 83/96 2026-02-20T11:55:48.641 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:48.641 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-20T11:55:48.641 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-20T11:55:48.641 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-20T11:55:48.641 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:48.701 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 84/96 2026-02-20T11:55:48.703 INFO:teuthology.orchestra.run.trial183.stdout: Installing : cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 84/96 2026-02-20T11:55:48.712 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9 85/96 2026-02-20T11:55:48.736 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el 86/96 2026-02-20T11:55:48.738 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 87/96 2026-02-20T11:55:51.638 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 87/96 2026-02-20T11:55:51.646 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 88/96 2026-02-20T11:55:52.100 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 88/96 2026-02-20T11:55:52.101 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 89/96 2026-02-20T11:55:52.114 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 89/96 2026-02-20T11:55:52.114 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 90/96 2026-02-20T11:55:52.176 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 90/96 2026-02-20T11:55:52.216 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9. 91/96 2026-02-20T11:55:52.217 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 92/96 2026-02-20T11:55:52.238 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 92/96 2026-02-20T11:55:52.238 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:52.238 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-20T11:55:52.238 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-20T11:55:52.238 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-20T11:55:52.238 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:52.251 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 93/96 2026-02-20T11:55:52.465 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 93/96 2026-02-20T11:55:52.466 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 94/96 2026-02-20T11:55:52.488 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 94/96 2026-02-20T11:55:52.488 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:52.488 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-20T11:55:52.488 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-20T11:55:52.488 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-20T11:55:52.488 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:52.573 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 95/96 2026-02-20T11:55:52.594 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 95/96 2026-02-20T11:55:52.594 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:52.594 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-20T11:55:52.594 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-20T11:55:52.594 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-20T11:55:52.594 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:52.595 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 96/96 2026-02-20T11:55:52.601 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 96/96 2026-02-20T11:55:52.625 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 96/96 2026-02-20T11:55:53.980 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 96/96 2026-02-20T11:55:53.980 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/96 2026-02-20T11:55:53.980 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 2/96 2026-02-20T11:55:53.980 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 3/96 2026-02-20T11:55:53.980 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 4/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 5/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_6 6/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_6 7/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el 8/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarc 9/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noa 10/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c314 11/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noa 12/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9. 13/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 14/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9 15/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 16/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 17/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-20T11:55:53.981 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-20T11:55:53.982 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-20T11:55:53.983 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-grafana-dashboards-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mds-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-k8sevents-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-modules-core-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mon-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-prometheus-alerts-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:54.049 INFO:teuthology.orchestra.run.trial183.stdout: cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: lua-5.4.4-4.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-20T11:55:54.050 INFO:teuthology.orchestra.run.trial183.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-ply-3.11-14.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-20T11:55:54.051 INFO:teuthology.orchestra.run.trial183.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: unzip-6.0-59.el9.x86_64 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: zip-3.0-35.el9.x86_64 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:54.052 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:54.118 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-base 2026-02-20T11:55:54.428 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:06 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:54.492 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:54.511 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:54.512 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:55:54.512 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:54.534 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install cephadm 2026-02-20T11:55:54.841 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:06 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:54.906 INFO:teuthology.orchestra.run.trial183.stdout:Package cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:54.925 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:54.925 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:55:54.925 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:54.947 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-immutable-object-cache 2026-02-20T11:55:55.255 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:07 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:55.339 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:55.339 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:55:55.339 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-20T11:55:55.339 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:55:55.339 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:55:55.340 INFO:teuthology.orchestra.run.trial183.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 149 k 2026-02-20T11:55:55.340 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:55.340 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:55:55.340 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:55:55.340 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-20T11:55:55.340 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:55.340 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 149 k 2026-02-20T11:55:55.340 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 455 k 2026-02-20T11:55:55.340 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:55:55.804 INFO:teuthology.orchestra.run.trial183.stdout:ceph-immutable-object-cache-20.3.0-5357.g0c3148 322 kB/s | 149 kB 00:00 2026-02-20T11:55:55.804 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:55.804 INFO:teuthology.orchestra.run.trial183.stdout:Total 322 kB/s | 149 kB 00:00 2026-02-20T11:55:55.804 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:55:55.809 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:55:55.809 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:55:55.840 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:55:55.840 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:55:55.895 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:55:55.897 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:55:55.919 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:55:55.919 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:55:55.919 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-20T11:55:55.920 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:56.198 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca. 1/1 2026-02-20T11:55:56.198 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:56.198 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:55:56.198 INFO:teuthology.orchestra.run.trial183.stdout: ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:55:56.198 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:56.199 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:56.226 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr 2026-02-20T11:55:56.538 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:08 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:56.603 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:56.621 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:56.622 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:55:56.622 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:56.644 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-dashboard 2026-02-20T11:55:56.953 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:08 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:57.017 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:57.035 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:57.036 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:55:57.036 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:57.058 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-20T11:55:57.367 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:09 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:57.432 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:57.450 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:57.451 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:55:57.451 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:57.473 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-rook 2026-02-20T11:55:57.787 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:09 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:57.852 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:57.871 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:57.871 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:55:57.872 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:57.893 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-cephadm 2026-02-20T11:55:58.206 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:10 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:58.270 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:55:58.288 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:58.288 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:55:58.289 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:58.309 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd 2026-02-20T11:55:58.617 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:10 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:58.681 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:58.700 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:58.700 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:55:58.701 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:58.722 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd-classic 2026-02-20T11:55:59.030 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:11 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:59.094 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:55:59.112 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:59.113 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:55:59.113 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:55:59.134 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-fuse 2026-02-20T11:55:59.440 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:11 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:55:59.523 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:55:59.524 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:55:59.524 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-20T11:55:59.525 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:55:59.525 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:55:59.525 INFO:teuthology.orchestra.run.trial183.stdout: ceph-fuse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 914 k 2026-02-20T11:55:59.525 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:59.525 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:55:59.525 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:55:59.525 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-20T11:55:59.525 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:55:59.525 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 914 k 2026-02-20T11:55:59.525 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 2.9 M 2026-02-20T11:55:59.526 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:55:59.996 INFO:teuthology.orchestra.run.trial183.stdout:ceph-fuse-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 1.9 MB/s | 914 kB 00:00 2026-02-20T11:55:59.997 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:55:59.997 INFO:teuthology.orchestra.run.trial183.stdout:Total 1.9 MB/s | 914 kB 00:00 2026-02-20T11:55:59.997 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:56:00.002 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:56:00.002 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:56:00.035 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:56:00.035 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:56:00.125 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:56:00.142 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:00.375 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:00.432 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:00.432 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:00.432 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:56:00.432 INFO:teuthology.orchestra.run.trial183.stdout: ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:56:00.432 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:00.432 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:00.459 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-volume 2026-02-20T11:56:00.769 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:12 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:00.834 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:56:00.852 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:00.853 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:56:00.853 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:00.874 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librados-devel 2026-02-20T11:56:01.190 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:13 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:01.272 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:01.273 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:01.273 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-20T11:56:01.273 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:01.273 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:56:01.273 INFO:teuthology.orchestra.run.trial183.stdout: librados-devel x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 124 k 2026-02-20T11:56:01.274 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:01.274 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:56:01.274 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:01.274 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-20T11:56:01.274 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:01.274 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 124 k 2026-02-20T11:56:01.274 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 434 k 2026-02-20T11:56:01.274 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:56:01.489 INFO:teuthology.orchestra.run.trial183.stdout:librados-devel-20.3.0-5357.g0c3148ca.el9.x86_64 575 kB/s | 124 kB 00:00 2026-02-20T11:56:01.489 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:56:01.489 INFO:teuthology.orchestra.run.trial183.stdout:Total 572 kB/s | 124 kB 00:00 2026-02-20T11:56:01.489 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:56:01.492 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:56:01.492 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:56:01.527 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:56:01.527 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:56:01.584 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:56:01.600 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:01.683 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:01.741 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:01.741 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:01.741 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:56:01.741 INFO:teuthology.orchestra.run.trial183.stdout: librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:56:01.741 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:01.741 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:01.768 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install libcephfs2 2026-02-20T11:56:02.076 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:14 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:02.146 INFO:teuthology.orchestra.run.trial183.stdout:Package libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:56:02.164 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:02.164 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:56:02.164 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:02.185 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install libcephfs-devel 2026-02-20T11:56:02.498 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:14 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:02.581 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:02.581 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:02.581 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-20T11:56:02.581 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:02.581 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:56:02.581 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-devel x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 37 k 2026-02-20T11:56:02.581 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:02.581 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:56:02.582 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:02.582 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-20T11:56:02.582 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:02.582 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 37 k 2026-02-20T11:56:02.582 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 171 k 2026-02-20T11:56:02.582 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:56:02.742 INFO:teuthology.orchestra.run.trial183.stdout:libcephfs-devel-20.3.0-5357.g0c3148ca.el9.x86_6 232 kB/s | 37 kB 00:00 2026-02-20T11:56:02.742 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:56:02.742 INFO:teuthology.orchestra.run.trial183.stdout:Total 230 kB/s | 37 kB 00:00 2026-02-20T11:56:02.743 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:56:02.745 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:56:02.745 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:56:02.748 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:56:02.748 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:56:02.769 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:56:02.785 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:02.841 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:02.898 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:02.898 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:02.898 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:56:02.898 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:56:02.899 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:02.899 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:02.926 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librados2 2026-02-20T11:56:03.234 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:15 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:03.298 INFO:teuthology.orchestra.run.trial183.stdout:Package librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:56:03.317 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:03.317 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:56:03.317 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:03.339 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librbd1 2026-02-20T11:56:03.653 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:15 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:03.717 INFO:teuthology.orchestra.run.trial183.stdout:Package librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:56:03.736 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:03.737 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:56:03.737 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:03.758 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rados 2026-02-20T11:56:04.064 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:16 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:04.129 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:56:04.148 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:04.148 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:56:04.148 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:04.171 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rgw 2026-02-20T11:56:04.481 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:16 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:04.546 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:56:04.564 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:04.565 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:56:04.565 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:04.587 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-cephfs 2026-02-20T11:56:04.897 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:16 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:04.961 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:56:04.979 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:04.980 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:56:04.980 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:05.002 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rbd 2026-02-20T11:56:05.313 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:17 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:05.378 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:56:05.396 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:05.397 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:56:05.397 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:05.418 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-fuse 2026-02-20T11:56:05.731 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:17 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:05.815 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:05.815 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:05.815 INFO:teuthology.orchestra.run.trial183.stdout: Package Architecture Version Repository Size 2026-02-20T11:56:05.815 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:05.816 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:56:05.816 INFO:teuthology.orchestra.run.trial183.stdout: rbd-fuse x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 86 k 2026-02-20T11:56:05.816 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:05.816 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:56:05.816 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:05.816 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-20T11:56:05.816 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:05.816 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 86 k 2026-02-20T11:56:05.816 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 236 k 2026-02-20T11:56:05.816 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:56:05.974 INFO:teuthology.orchestra.run.trial183.stdout:rbd-fuse-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 543 kB/s | 86 kB 00:00 2026-02-20T11:56:05.976 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:56:05.976 INFO:teuthology.orchestra.run.trial183.stdout:Total 535 kB/s | 86 kB 00:00 2026-02-20T11:56:05.976 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:56:05.981 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:56:05.981 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:56:06.012 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:56:06.012 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:56:06.063 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:56:06.080 INFO:teuthology.orchestra.run.trial183.stdout: Installing : rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:06.164 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:06.223 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:06.223 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:06.223 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:56:06.223 INFO:teuthology.orchestra.run.trial183.stdout: rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:56:06.223 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:06.223 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:06.251 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-mirror 2026-02-20T11:56:06.565 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:18 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:06.648 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:06.648 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout: rbd-mirror x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 2.6 M 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 2.6 M 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 11 M 2026-02-20T11:56:06.649 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:56:07.480 INFO:teuthology.orchestra.run.trial183.stdout:rbd-mirror-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 3.1 MB/s | 2.6 MB 00:00 2026-02-20T11:56:07.481 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:56:07.481 INFO:teuthology.orchestra.run.trial183.stdout:Total 3.1 MB/s | 2.6 MB 00:00 2026-02-20T11:56:07.481 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:56:07.485 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:56:07.486 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:56:07.525 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:56:07.525 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:56:07.689 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:56:07.690 INFO:teuthology.orchestra.run.trial183.stdout: Installing : rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:07.714 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:07.714 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-20T11:56:07.714 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-20T11:56:07.714 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T11:56:07.714 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T11:56:07.714 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:08.007 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:08.007 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:08.008 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:56:08.008 INFO:teuthology.orchestra.run.trial183.stdout: rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:56:08.008 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:08.008 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:08.036 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-nbd 2026-02-20T11:56:08.348 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:20 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:08.431 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:08.432 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:08.432 INFO:teuthology.orchestra.run.trial183.stdout: Package Architecture Version Repository Size 2026-02-20T11:56:08.432 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:08.432 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:56:08.433 INFO:teuthology.orchestra.run.trial183.stdout: rbd-nbd x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 167 k 2026-02-20T11:56:08.433 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:08.433 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:56:08.433 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:08.433 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-20T11:56:08.433 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:08.433 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 167 k 2026-02-20T11:56:08.433 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 496 k 2026-02-20T11:56:08.433 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:56:08.864 INFO:teuthology.orchestra.run.trial183.stdout:rbd-nbd-20.3.0-5357.g0c3148ca.el9.x86_64.rpm 386 kB/s | 167 kB 00:00 2026-02-20T11:56:08.865 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:56:08.865 INFO:teuthology.orchestra.run.trial183.stdout:Total 385 kB/s | 167 kB 00:00 2026-02-20T11:56:08.865 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:56:08.869 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:56:08.869 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:56:08.903 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:56:08.903 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:56:08.961 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:56:08.978 INFO:teuthology.orchestra.run.trial183.stdout: Installing : rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:09.061 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:09.119 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/1 2026-02-20T11:56:09.119 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:09.119 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:56:09.119 INFO:teuthology.orchestra.run.trial183.stdout: rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:56:09.119 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:09.119 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:09.147 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd-crimson 2026-02-20T11:56:09.466 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:21 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:09.530 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-crimson-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:56:09.548 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:09.549 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:56:09.549 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:56:09.571 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-20T11:56:09.884 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:02:21 ago on Fri 20 Feb 2026 11:53:48 AM UTC. 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 433 M 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout:Installing weak dependencies: 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout: ceph-debugsource x86_64 2:20.3.0-5357.g0c3148ca.el9 ceph 12 M 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout:Install 2 Packages 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:56:09.967 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 445 M 2026-02-20T11:56:09.968 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 2.7 G 2026-02-20T11:56:09.968 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-20T11:56:12.989 INFO:teuthology.orchestra.run.trial183.stdout:(1/2): ceph-debugsource-20.3.0-5357.g0c3148ca.e 4.0 MB/s | 12 MB 00:03 2026-02-20T11:56:37.145 INFO:teuthology.orchestra.run.trial153.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5357.g 7.7 MB/s | 433 MB 00:56 2026-02-20T11:56:37.146 INFO:teuthology.orchestra.run.trial153.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:56:37.147 INFO:teuthology.orchestra.run.trial153.stdout:Total 7.9 MB/s | 445 MB 00:56 2026-02-20T11:56:37.147 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction check 2026-02-20T11:56:37.151 INFO:teuthology.orchestra.run.trial153.stdout:Transaction check succeeded. 2026-02-20T11:56:37.151 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction test 2026-02-20T11:56:38.529 INFO:teuthology.orchestra.run.trial153.stdout:Transaction test succeeded. 2026-02-20T11:56:38.529 INFO:teuthology.orchestra.run.trial153.stdout:Running transaction 2026-02-20T11:56:42.148 INFO:teuthology.orchestra.run.trial153.stdout: Preparing : 1/1 2026-02-20T11:56:50.368 INFO:teuthology.orchestra.run.trial183.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5357.g 11 MB/s | 433 MB 00:40 2026-02-20T11:56:50.370 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-20T11:56:50.370 INFO:teuthology.orchestra.run.trial183.stdout:Total 11 MB/s | 445 MB 00:40 2026-02-20T11:56:50.370 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-20T11:56:50.375 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-20T11:56:50.375 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-20T11:56:51.745 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-20T11:56:51.745 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-20T11:56:55.339 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-20T11:57:04.527 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/2 2026-02-20T11:57:04.544 INFO:teuthology.orchestra.run.trial153.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:57:06.534 INFO:teuthology.orchestra.run.trial153.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:57:06.534 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/2 2026-02-20T11:57:06.592 INFO:teuthology.orchestra.run.trial153.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:57:06.592 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:57:06.593 INFO:teuthology.orchestra.run.trial153.stdout:Installed: 2026-02-20T11:57:06.593 INFO:teuthology.orchestra.run.trial153.stdout: ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:57:06.593 INFO:teuthology.orchestra.run.trial153.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:57:06.593 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:57:06.593 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:57:06.664 DEBUG:teuthology.parallel:result is None 2026-02-20T11:57:17.630 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/2 2026-02-20T11:57:17.649 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:57:19.559 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:57:19.560 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 1/2 2026-02-20T11:57:19.618 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.e 2/2 2026-02-20T11:57:19.618 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:57:19.618 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-20T11:57:19.618 INFO:teuthology.orchestra.run.trial183.stdout: ceph-debugsource-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:57:19.618 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5357.g0c3148ca.el9.x86_64 2026-02-20T11:57:19.618 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:57:19.619 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:19.692 DEBUG:teuthology.parallel:result is None 2026-02-20T11:57:19.692 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:57:19.856 DEBUG:teuthology.orchestra.run.trial153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-20T11:57:19.878 INFO:teuthology.orchestra.run.trial153.stdout:20.3.0-5357.g0c3148ca.el9 2026-02-20T11:57:19.879 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5357.g0c3148ca.el9 2026-02-20T11:57:19.879 INFO:teuthology.task.install:The correct ceph version 20.3.0-5357.g0c3148ca is installed. 2026-02-20T11:57:19.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:57:20.029 DEBUG:teuthology.orchestra.run.trial183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-20T11:57:20.050 INFO:teuthology.orchestra.run.trial183.stdout:20.3.0-5357.g0c3148ca.el9 2026-02-20T11:57:20.051 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5357.g0c3148ca.el9 2026-02-20T11:57:20.051 INFO:teuthology.task.install:The correct ceph version 20.3.0-5357.g0c3148ca is installed. 2026-02-20T11:57:20.053 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-20T11:57:20.053 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:57:20.054 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-20T11:57:20.079 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:57:20.080 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-20T11:57:20.116 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-20T11:57:20.116 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:57:20.116 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/usr/bin/daemon-helper 2026-02-20T11:57:20.145 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-20T11:57:20.209 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:57:20.209 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/usr/bin/daemon-helper 2026-02-20T11:57:20.235 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-20T11:57:20.299 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-20T11:57:20.299 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:57:20.299 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-20T11:57:20.326 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-20T11:57:20.390 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:57:20.390 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-20T11:57:20.416 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-20T11:57:20.480 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-20T11:57:20.481 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:57:20.481 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/usr/bin/stdin-killer 2026-02-20T11:57:20.507 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-20T11:57:20.572 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:57:20.573 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/usr/bin/stdin-killer 2026-02-20T11:57:20.597 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-20T11:57:20.662 INFO:teuthology.run_tasks:Running task pexec... 2026-02-20T11:57:20.680 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-20T11:57:20.681 DEBUG:teuthology.orchestra.run.trial153:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-20T11:57:20.681 DEBUG:teuthology.orchestra.run.trial183:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-20T11:57:20.684 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial153.front.sepia.ceph.com 2026-02-20T11:57:20.684 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-20T11:57:20.704 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial183.front.sepia.ceph.com 2026-02-20T11:57:20.704 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-20T11:57:20.735 DEBUG:teuthology.parallel:result is None 2026-02-20T11:57:20.754 DEBUG:teuthology.parallel:result is None 2026-02-20T11:57:20.755 INFO:teuthology.run_tasks:Running task install... 2026-02-20T11:57:20.769 DEBUG:teuthology.task.install:project ceph 2026-02-20T11:57:20.769 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}} 2026-02-20T11:57:20.769 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'} 2026-02-20T11:57:20.769 INFO:teuthology.task.install:Using flavor: debug 2026-02-20T11:57:20.778 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-20T11:57:20.778 INFO:teuthology.task.install:extra packages: [] 2026-02-20T11:57:20.778 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'tag': None, 'wait_for_package': False} 2026-02-20T11:57:20.779 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:57:20.780 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'tag': None, 'wait_for_package': False} 2026-02-20T11:57:20.781 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:57:20.970 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/ 2026-02-20T11:57:20.971 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5357.g0c3148ca 2026-02-20T11:57:20.982 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/ 2026-02-20T11:57:20.982 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5357.g0c3148ca 2026-02-20T11:57:21.237 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-20T11:57:21.237 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:57:21.237 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-20T11:57:21.263 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-20T11:57:21.263 DEBUG:teuthology.orchestra.run.trial153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-20T11:57:21.281 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-perezjos-crimson-only-19-02-2026-PR64439/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-20T11:57:21.281 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:57:21.281 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-20T11:57:21.307 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-20T11:57:21.308 DEBUG:teuthology.orchestra.run.trial183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0c3148ca19c827ca2a7a9b92c2554d99507fd1f4/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-20T11:57:21.330 DEBUG:teuthology.orchestra.run.trial153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-20T11:57:21.374 DEBUG:teuthology.orchestra.run.trial183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-20T11:57:21.394 DEBUG:teuthology.orchestra.run.trial153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-20T11:57:21.439 DEBUG:teuthology.orchestra.run.trial183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-20T11:57:21.450 INFO:teuthology.orchestra.run.trial153.stdout:check_obsoletes = 1 2026-02-20T11:57:21.460 DEBUG:teuthology.orchestra.run.trial153:> sudo yum clean all 2026-02-20T11:57:21.494 INFO:teuthology.orchestra.run.trial183.stdout:check_obsoletes = 1 2026-02-20T11:57:21.504 DEBUG:teuthology.orchestra.run.trial183:> sudo yum clean all 2026-02-20T11:57:21.622 INFO:teuthology.orchestra.run.trial153.stdout:56 files removed 2026-02-20T11:57:21.637 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-radosgw 2026-02-20T11:57:21.668 INFO:teuthology.orchestra.run.trial183.stdout:56 files removed 2026-02-20T11:57:21.683 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-radosgw 2026-02-20T11:57:22.231 INFO:teuthology.orchestra.run.trial153.stdout:ceph packages for x86_64 211 kB/s | 94 kB 00:00 2026-02-20T11:57:22.281 INFO:teuthology.orchestra.run.trial183.stdout:ceph packages for x86_64 210 kB/s | 94 kB 00:00 2026-02-20T11:57:22.658 INFO:teuthology.orchestra.run.trial153.stdout:ceph noarch packages 61 kB/s | 24 kB 00:00 2026-02-20T11:57:22.706 INFO:teuthology.orchestra.run.trial183.stdout:ceph noarch packages 61 kB/s | 24 kB 00:00 2026-02-20T11:57:23.049 INFO:teuthology.orchestra.run.trial153.stdout:ceph source packages 5.6 kB/s | 2.1 kB 00:00 2026-02-20T11:57:23.098 INFO:teuthology.orchestra.run.trial183.stdout:ceph source packages 5.6 kB/s | 2.1 kB 00:00 2026-02-20T11:57:23.603 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.9 MB 00:00 2026-02-20T11:57:26.880 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.9 MB 00:03 2026-02-20T11:57:27.337 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - AppStream 8.4 MB/s | 27 MB 00:03 2026-02-20T11:57:30.705 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 7.8 MB 00:00 2026-02-20T11:57:32.339 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 20 kB 00:00 2026-02-20T11:57:33.656 INFO:teuthology.orchestra.run.trial183.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2026-02-20T11:57:36.850 INFO:teuthology.orchestra.run.trial183.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-20T11:57:38.020 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:38.039 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:38.040 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:38.040 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:38.064 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-test 2026-02-20T11:57:38.374 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:38.438 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:38.457 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:38.457 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:38.458 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:38.479 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph 2026-02-20T11:57:38.786 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:38.851 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:38.870 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:38.871 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:38.871 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:38.892 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-base 2026-02-20T11:57:39.200 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:39.265 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:39.283 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:39.284 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:39.284 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:39.306 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install cephadm 2026-02-20T11:57:39.616 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:39.681 INFO:teuthology.orchestra.run.trial183.stdout:Package cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:57:39.699 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:39.700 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:39.700 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:39.721 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-immutable-object-cache 2026-02-20T11:57:40.038 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:40.103 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:40.121 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:40.122 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:40.122 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:40.144 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr 2026-02-20T11:57:40.456 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:40.520 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:40.539 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:40.539 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:40.539 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:40.561 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-dashboard 2026-02-20T11:57:40.872 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:40.936 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:57:40.954 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:40.955 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:40.955 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:40.976 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-20T11:57:41.288 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:41.353 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:57:41.371 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:41.372 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:41.372 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:41.395 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-rook 2026-02-20T11:57:41.703 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:41.768 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:57:41.787 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:41.787 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:41.788 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:41.810 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-cephadm 2026-02-20T11:57:42.118 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:42.183 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:57:42.201 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:42.202 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:42.202 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:42.224 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd 2026-02-20T11:57:42.535 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:42.600 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:42.619 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:42.619 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:42.619 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:42.641 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd-classic 2026-02-20T11:57:42.954 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:43.019 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:43.037 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:43.038 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:43.038 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:43.060 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-fuse 2026-02-20T11:57:43.369 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:43.434 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:43.453 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:43.453 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:43.454 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:43.475 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-volume 2026-02-20T11:57:43.785 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:43.850 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:57:43.869 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:43.869 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:43.869 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:43.891 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librados-devel 2026-02-20T11:57:44.201 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:08 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:44.266 INFO:teuthology.orchestra.run.trial183.stdout:Package librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:44.284 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:44.285 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:44.285 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:44.306 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install libcephfs2 2026-02-20T11:57:44.617 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:08 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:44.682 INFO:teuthology.orchestra.run.trial183.stdout:Package libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:44.701 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:44.701 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:44.701 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:44.722 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install libcephfs-devel 2026-02-20T11:57:45.033 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:45.098 INFO:teuthology.orchestra.run.trial183.stdout:Package libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:45.116 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:45.117 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:45.117 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:45.139 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librados2 2026-02-20T11:57:45.447 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:45.511 INFO:teuthology.orchestra.run.trial183.stdout:Package librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:45.530 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:45.531 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:45.531 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:45.552 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librbd1 2026-02-20T11:57:45.857 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:45.921 INFO:teuthology.orchestra.run.trial183.stdout:Package librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:45.940 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:45.940 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:45.941 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:45.962 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rados 2026-02-20T11:57:46.273 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:46.308 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 27 MB 00:18 2026-02-20T11:57:46.338 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:46.357 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:46.358 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:46.358 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:46.381 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rgw 2026-02-20T11:57:46.689 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:46.755 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:46.774 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:46.774 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:46.775 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:46.796 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-cephfs 2026-02-20T11:57:47.107 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:47.172 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:47.191 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:47.191 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:47.192 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:47.215 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rbd 2026-02-20T11:57:47.526 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:47.591 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:47.610 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:47.611 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:47.611 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:47.633 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-fuse 2026-02-20T11:57:47.944 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:48.009 INFO:teuthology.orchestra.run.trial183.stdout:Package rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:48.027 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:48.028 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:48.028 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:48.050 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-mirror 2026-02-20T11:57:48.360 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:12 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:48.425 INFO:teuthology.orchestra.run.trial183.stdout:Package rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:48.444 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:48.445 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:48.445 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:48.466 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-nbd 2026-02-20T11:57:48.772 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:12 ago on Fri 20 Feb 2026 11:57:36 AM UTC. 2026-02-20T11:57:48.837 INFO:teuthology.orchestra.run.trial183.stdout:Package rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:48.855 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-20T11:57:48.856 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-20T11:57:48.856 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-20T11:57:48.882 DEBUG:teuthology.parallel:result is None 2026-02-20T11:57:50.054 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - CRB 5.6 MB/s | 7.8 MB 00:01 2026-02-20T11:57:52.004 INFO:teuthology.orchestra.run.trial153.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 20 kB 00:01 2026-02-20T11:57:52.812 INFO:teuthology.orchestra.run.trial153.stdout:Extra Packages for Enterprise Linux 27 MB/s | 20 MB 00:00 2026-02-20T11:57:55.967 INFO:teuthology.orchestra.run.trial153.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-20T11:57:57.143 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-radosgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:57.162 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:57:57.162 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:57:57.163 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:57:57.186 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-test 2026-02-20T11:57:57.494 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:57:57.558 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-test-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:57.577 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:57:57.578 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:57:57.578 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:57:57.599 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph 2026-02-20T11:57:57.908 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:02 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:57:57.973 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:57.992 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:57:57.993 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:57:57.993 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:57:58.014 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-base 2026-02-20T11:57:58.324 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:57:58.389 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-base-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:58.408 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:57:58.408 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:57:58.408 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:57:58.430 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install cephadm 2026-02-20T11:57:58.743 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:03 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:57:58.809 INFO:teuthology.orchestra.run.trial153.stdout:Package cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:57:58.828 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:57:58.828 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:57:58.828 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:57:58.850 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-immutable-object-cache 2026-02-20T11:57:59.160 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:57:59.224 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-immutable-object-cache-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:59.243 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:57:59.243 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:57:59.243 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:57:59.264 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr 2026-02-20T11:57:59.576 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:57:59.641 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:57:59.660 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:57:59.660 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:57:59.661 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:57:59.682 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-dashboard 2026-02-20T11:57:59.995 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:04 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:00.059 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-dashboard-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:58:00.078 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:00.078 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:00.078 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:00.100 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-20T11:58:00.415 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:00.480 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:58:00.498 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:00.498 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:00.498 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:00.520 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-rook 2026-02-20T11:58:00.827 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:05 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:00.894 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-rook-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:58:00.913 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:00.913 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:00.914 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:00.935 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-mgr-cephadm 2026-02-20T11:58:01.249 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:01.315 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-mgr-cephadm-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:58:01.333 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:01.334 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:01.334 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:01.356 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-osd 2026-02-20T11:58:01.664 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:06 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:01.730 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-osd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:01.748 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:01.749 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:01.749 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:01.771 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-osd-classic 2026-02-20T11:58:02.086 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:02.152 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-osd-classic-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:02.170 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:02.171 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:02.171 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:02.193 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-fuse 2026-02-20T11:58:02.505 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:02.570 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:02.589 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:02.590 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:02.590 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:02.611 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install ceph-volume 2026-02-20T11:58:02.919 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:07 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:02.985 INFO:teuthology.orchestra.run.trial153.stdout:Package ceph-volume-2:20.3.0-5357.g0c3148ca.el9.noarch is already installed. 2026-02-20T11:58:03.003 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:03.004 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:03.004 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:03.026 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install librados-devel 2026-02-20T11:58:03.343 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:08 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:03.409 INFO:teuthology.orchestra.run.trial153.stdout:Package librados-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:03.428 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:03.428 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:03.429 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:03.451 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install libcephfs2 2026-02-20T11:58:03.763 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:08 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:03.828 INFO:teuthology.orchestra.run.trial153.stdout:Package libcephfs2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:03.847 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:03.848 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:03.848 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:03.869 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install libcephfs-devel 2026-02-20T11:58:04.182 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:04.249 INFO:teuthology.orchestra.run.trial153.stdout:Package libcephfs-devel-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:04.267 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:04.268 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:04.268 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:04.290 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install librados2 2026-02-20T11:58:04.596 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:09 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:04.661 INFO:teuthology.orchestra.run.trial153.stdout:Package librados2-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:04.680 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:04.690 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:04.691 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:04.701 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install librbd1 2026-02-20T11:58:05.013 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:05.079 INFO:teuthology.orchestra.run.trial153.stdout:Package librbd1-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:05.098 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:05.098 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:05.098 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:05.120 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-rados 2026-02-20T11:58:05.431 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:05.499 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-rados-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:05.517 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:05.518 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:05.518 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:05.541 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-rgw 2026-02-20T11:58:05.853 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:10 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:05.919 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-rgw-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:05.938 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:05.938 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:05.938 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:05.960 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-cephfs 2026-02-20T11:58:06.273 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:06.339 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-cephfs-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:06.357 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:06.358 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:06.358 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:06.380 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install python3-rbd 2026-02-20T11:58:06.699 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:11 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:06.765 INFO:teuthology.orchestra.run.trial153.stdout:Package python3-rbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:06.783 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:06.784 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:06.784 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:06.805 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install rbd-fuse 2026-02-20T11:58:07.118 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:12 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:07.184 INFO:teuthology.orchestra.run.trial153.stdout:Package rbd-fuse-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:07.202 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:07.203 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:07.203 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:07.224 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install rbd-mirror 2026-02-20T11:58:07.535 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:12 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:07.602 INFO:teuthology.orchestra.run.trial153.stdout:Package rbd-mirror-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:07.620 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:07.621 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:07.621 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:07.642 DEBUG:teuthology.orchestra.run.trial153:> sudo yum -y install rbd-nbd 2026-02-20T11:58:07.952 INFO:teuthology.orchestra.run.trial153.stdout:Last metadata expiration check: 0:00:12 ago on Fri 20 Feb 2026 11:57:55 AM UTC. 2026-02-20T11:58:08.017 INFO:teuthology.orchestra.run.trial153.stdout:Package rbd-nbd-2:20.3.0-5357.g0c3148ca.el9.x86_64 is already installed. 2026-02-20T11:58:08.036 INFO:teuthology.orchestra.run.trial153.stdout:Dependencies resolved. 2026-02-20T11:58:08.036 INFO:teuthology.orchestra.run.trial153.stdout:Nothing to do. 2026-02-20T11:58:08.037 INFO:teuthology.orchestra.run.trial153.stdout:Complete! 2026-02-20T11:58:08.062 DEBUG:teuthology.parallel:result is None 2026-02-20T11:58:08.063 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:58:08.226 DEBUG:teuthology.orchestra.run.trial153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-20T11:58:08.246 INFO:teuthology.orchestra.run.trial153.stdout:20.3.0-5357.g0c3148ca.el9 2026-02-20T11:58:08.246 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5357.g0c3148ca.el9 2026-02-20T11:58:08.246 INFO:teuthology.task.install:The correct ceph version 20.3.0-5357.g0c3148ca is installed. 2026-02-20T11:58:08.248 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=0c3148ca19c827ca2a7a9b92c2554d99507fd1f4 2026-02-20T11:58:08.387 DEBUG:teuthology.orchestra.run.trial183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-20T11:58:08.408 INFO:teuthology.orchestra.run.trial183.stdout:20.3.0-5357.g0c3148ca.el9 2026-02-20T11:58:08.409 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5357.g0c3148ca.el9 2026-02-20T11:58:08.409 INFO:teuthology.task.install:The correct ceph version 20.3.0-5357.g0c3148ca is installed. 2026-02-20T11:58:08.410 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-20T11:58:08.410 INFO:teuthology.run_tasks:Running task ceph... 2026-02-20T11:58:08.501 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-20T11:58:08.501 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod 777 /var/log/ceph 2026-02-20T11:58:08.503 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod 777 /var/log/ceph 2026-02-20T11:58:08.529 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-20T11:58:08.529 DEBUG:teuthology.orchestra.run.trial153:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-20T11:58:08.568 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-20T11:58:08.592 INFO:tasks.ceph:Creating extra log directories... 2026-02-20T11:58:08.593 DEBUG:teuthology.orchestra.run.trial153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-20T11:58:08.634 DEBUG:teuthology.orchestra.run.trial183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-20T11:58:08.672 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-20T11:58:08.672 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-20T11:58:08.673 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590', 'branch': 'wip-perezjos-crimson-only-19-02-2026-PR64439', 'description': 'crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} thrashers/{default} thrashosds-health workloads/snaps-few-objects}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '60590', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-perezjos-crimson-only-19-02-2026-PR64439'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '445d5c9ba18a4040e70e25303230c36c77773acb'}}, 'owner': 'scheduled_jjperez@soko04.front.sepia.ceph.com', 'priority': 101, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 1736, 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '445d5c9ba18a4040e70e25303230c36c77773acb', 'targets': {'trial153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOiFzemx1T/+ABLF5v0t5hoKbNWa4IQRyZtGZE4xgH2BHebCKfBoD+PQGjP2XPt+sdgvKtz4qcRVhJz4PcsaIcc=', 'trial183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCZAv4U83dWpuIrSXfTRzNldrJbHj71xsfPntOMn5d6Jq/rvPqvFoMpv52jgAX28bKUEVLeFT/9KCWOjkrAktQU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'reimage-ssh-down', 'teuthology_sha1': 'e6d9099ce03dc21193fda653159ec82465c3a94a', 'timestamp': '2026-02-20_10:21:31', 'tube': 'trial', 'user': 'jjperez', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699'} 2026-02-20T11:58:08.673 DEBUG:teuthology.orchestra.run.trial153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-20T11:58:08.700 DEBUG:teuthology.orchestra.run.trial183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-20T11:58:08.717 DEBUG:teuthology.orchestra.run.trial153:> sudo install -d -m0777 -- /var/run/ceph 2026-02-20T11:58:08.758 DEBUG:teuthology.orchestra.run.trial183:> sudo install -d -m0777 -- /var/run/ceph 2026-02-20T11:58:08.783 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:08.783 DEBUG:teuthology.orchestra.run.trial153:> dd if=/scratch_devs of=/dev/stdout 2026-02-20T11:58:08.837 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-20T11:58:08.838 DEBUG:teuthology.orchestra.run.trial153:> stat /dev/vg_nvme/lv_1 2026-02-20T11:58:08.894 INFO:teuthology.orchestra.run.trial153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-20T11:58:08.894 INFO:teuthology.orchestra.run.trial153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:58:08.894 INFO:teuthology.orchestra.run.trial153.stdout:Device: 6h/6d Inode: 1306 Links: 1 2026-02-20T11:58:08.894 INFO:teuthology.orchestra.run.trial153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:58:08.895 INFO:teuthology.orchestra.run.trial153.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:58:08.895 INFO:teuthology.orchestra.run.trial153.stdout:Access: 2026-02-20 11:55:39.087583688 +0000 2026-02-20T11:58:08.895 INFO:teuthology.orchestra.run.trial153.stdout:Modify: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:08.895 INFO:teuthology.orchestra.run.trial153.stdout:Change: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:08.895 INFO:teuthology.orchestra.run.trial153.stdout: Birth: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:08.895 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-20T11:58:08.957 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records in 2026-02-20T11:58:08.958 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records out 2026-02-20T11:58:08.958 INFO:teuthology.orchestra.run.trial153.stderr:512 bytes copied, 0.000179382 s, 2.9 MB/s 2026-02-20T11:58:08.959 DEBUG:teuthology.orchestra.run.trial153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-20T11:58:09.018 DEBUG:teuthology.orchestra.run.trial153:> stat /dev/vg_nvme/lv_2 2026-02-20T11:58:09.074 INFO:teuthology.orchestra.run.trial153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-20T11:58:09.074 INFO:teuthology.orchestra.run.trial153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:58:09.075 INFO:teuthology.orchestra.run.trial153.stdout:Device: 6h/6d Inode: 1299 Links: 1 2026-02-20T11:58:09.075 INFO:teuthology.orchestra.run.trial153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:58:09.075 INFO:teuthology.orchestra.run.trial153.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:58:09.075 INFO:teuthology.orchestra.run.trial153.stdout:Access: 2026-02-20 11:55:39.088583650 +0000 2026-02-20T11:58:09.075 INFO:teuthology.orchestra.run.trial153.stdout:Modify: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:09.075 INFO:teuthology.orchestra.run.trial153.stdout:Change: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:09.075 INFO:teuthology.orchestra.run.trial153.stdout: Birth: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:09.075 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-20T11:58:09.139 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records in 2026-02-20T11:58:09.139 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records out 2026-02-20T11:58:09.139 INFO:teuthology.orchestra.run.trial153.stderr:512 bytes copied, 0.000141419 s, 3.6 MB/s 2026-02-20T11:58:09.140 DEBUG:teuthology.orchestra.run.trial153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-20T11:58:09.198 DEBUG:teuthology.orchestra.run.trial153:> stat /dev/vg_nvme/lv_3 2026-02-20T11:58:09.254 INFO:teuthology.orchestra.run.trial153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-20T11:58:09.254 INFO:teuthology.orchestra.run.trial153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:58:09.254 INFO:teuthology.orchestra.run.trial153.stdout:Device: 6h/6d Inode: 1301 Links: 1 2026-02-20T11:58:09.254 INFO:teuthology.orchestra.run.trial153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:58:09.254 INFO:teuthology.orchestra.run.trial153.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:58:09.254 INFO:teuthology.orchestra.run.trial153.stdout:Access: 2026-02-20 11:55:39.088583650 +0000 2026-02-20T11:58:09.255 INFO:teuthology.orchestra.run.trial153.stdout:Modify: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:09.255 INFO:teuthology.orchestra.run.trial153.stdout:Change: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:09.255 INFO:teuthology.orchestra.run.trial153.stdout: Birth: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:09.255 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-20T11:58:09.317 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records in 2026-02-20T11:58:09.317 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records out 2026-02-20T11:58:09.317 INFO:teuthology.orchestra.run.trial153.stderr:512 bytes copied, 0.00018894 s, 2.7 MB/s 2026-02-20T11:58:09.319 DEBUG:teuthology.orchestra.run.trial153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-20T11:58:09.376 DEBUG:teuthology.orchestra.run.trial153:> stat /dev/vg_nvme/lv_4 2026-02-20T11:58:09.432 INFO:teuthology.orchestra.run.trial153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-20T11:58:09.432 INFO:teuthology.orchestra.run.trial153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:58:09.432 INFO:teuthology.orchestra.run.trial153.stdout:Device: 6h/6d Inode: 1285 Links: 1 2026-02-20T11:58:09.432 INFO:teuthology.orchestra.run.trial153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:58:09.432 INFO:teuthology.orchestra.run.trial153.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:58:09.432 INFO:teuthology.orchestra.run.trial153.stdout:Access: 2026-02-20 11:55:39.088583650 +0000 2026-02-20T11:58:09.432 INFO:teuthology.orchestra.run.trial153.stdout:Modify: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:09.433 INFO:teuthology.orchestra.run.trial153.stdout:Change: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:09.433 INFO:teuthology.orchestra.run.trial153.stdout: Birth: 2026-02-20 11:53:59.999545499 +0000 2026-02-20T11:58:09.433 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-20T11:58:09.494 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records in 2026-02-20T11:58:09.494 INFO:teuthology.orchestra.run.trial153.stderr:1+0 records out 2026-02-20T11:58:09.494 INFO:teuthology.orchestra.run.trial153.stderr:512 bytes copied, 0.000145929 s, 3.5 MB/s 2026-02-20T11:58:09.495 DEBUG:teuthology.orchestra.run.trial153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-20T11:58:09.555 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-20T11:58:09.555 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:58:09.555 DEBUG:teuthology.orchestra.run.trial183:> dd if=/scratch_devs of=/dev/stdout 2026-02-20T11:58:09.574 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-20T11:58:09.574 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_1 2026-02-20T11:58:09.630 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-20T11:58:09.631 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:58:09.631 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1289 Links: 1 2026-02-20T11:58:09.631 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:58:09.631 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:58:09.631 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-20 11:56:07.845495870 +0000 2026-02-20T11:58:09.631 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-20 11:53:58.122331153 +0000 2026-02-20T11:58:09.631 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-20 11:53:58.122331153 +0000 2026-02-20T11:58:09.631 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-20 11:53:58.122331153 +0000 2026-02-20T11:58:09.632 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-20T11:58:09.694 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-20T11:58:09.694 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-20T11:58:09.694 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000166499 s, 3.1 MB/s 2026-02-20T11:58:09.695 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-20T11:58:09.753 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_2 2026-02-20T11:58:09.809 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-20T11:58:09.809 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:58:09.809 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1298 Links: 1 2026-02-20T11:58:09.810 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:58:09.810 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:58:09.810 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-20 11:56:07.846495866 +0000 2026-02-20T11:58:09.810 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-20 11:53:58.123331118 +0000 2026-02-20T11:58:09.810 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-20 11:53:58.123331118 +0000 2026-02-20T11:58:09.810 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-20 11:53:58.123331118 +0000 2026-02-20T11:58:09.810 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-20T11:58:09.873 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-20T11:58:09.873 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-20T11:58:09.873 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000130713 s, 3.9 MB/s 2026-02-20T11:58:09.875 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-20T11:58:09.932 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_3 2026-02-20T11:58:09.988 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-20T11:58:09.989 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:58:09.989 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1310 Links: 1 2026-02-20T11:58:09.989 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:58:09.989 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:58:09.989 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-20 11:56:07.846495866 +0000 2026-02-20T11:58:09.989 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-20 11:53:58.124331082 +0000 2026-02-20T11:58:09.989 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-20 11:53:58.124331082 +0000 2026-02-20T11:58:09.989 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-20 11:53:58.124331082 +0000 2026-02-20T11:58:09.990 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-20T11:58:10.051 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-20T11:58:10.052 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-20T11:58:10.052 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000151983 s, 3.4 MB/s 2026-02-20T11:58:10.053 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-20T11:58:10.110 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_4 2026-02-20T11:58:10.167 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-20T11:58:10.168 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T11:58:10.168 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1305 Links: 1 2026-02-20T11:58:10.168 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T11:58:10.168 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-20T11:58:10.168 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-20 11:56:07.846495866 +0000 2026-02-20T11:58:10.168 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-20 11:53:58.123331118 +0000 2026-02-20T11:58:10.168 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-20 11:53:58.123331118 +0000 2026-02-20T11:58:10.168 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-20 11:53:58.123331118 +0000 2026-02-20T11:58:10.169 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-20T11:58:10.230 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-20T11:58:10.230 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-20T11:58:10.230 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.00014555 s, 3.5 MB/s 2026-02-20T11:58:10.231 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-20T11:58:10.288 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-20T11:58:10.289 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial183.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-20T11:58:10.289 INFO:tasks.ceph:Generating config... 2026-02-20T11:58:10.292 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-20T11:58:10.292 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-20T11:58:10.292 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-20T11:58:10.292 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-20T11:58:10.292 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-20T11:58:10.292 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-20T11:58:10.292 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-20T11:58:10.293 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-20T11:58:10.293 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-20T11:58:10.293 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-20T11:58:10.293 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-20T11:58:10.293 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-20T11:58:10.293 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-20T11:58:10.293 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-20T11:58:10.293 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-20T11:58:10.293 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-20T11:58:10.293 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-20T11:58:10.294 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-20T11:58:10.294 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-20T11:58:10.294 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2026-02-20T11:58:10.294 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2026-02-20T11:58:10.294 INFO:tasks.ceph:[osd] osd delete sleep = 1 2026-02-20T11:58:10.294 INFO:tasks.ceph:[osd] osd max backfills = 3 2026-02-20T11:58:10.294 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2026-02-20T11:58:10.294 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-20T11:58:10.294 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2026-02-20T11:58:10.294 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2026-02-20T11:58:10.295 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-20T11:58:10.295 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2026-02-20T11:58:10.295 INFO:tasks.ceph:[osd] osd target transaction size = 5 2026-02-20T11:58:10.295 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-20T11:58:10.295 INFO:tasks.ceph:[osd] seastore cachepin type = 2Q 2026-02-20T11:58:10.295 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-20T11:58:10.295 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-20T11:58:10.295 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-20T11:58:10.295 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-20T11:58:10.295 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-20T11:58:10.295 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-20T11:58:10.296 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-20T11:58:10.296 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-20T11:58:10.296 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 2 2026-02-20T11:58:10.296 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 1 2026-02-20T11:58:10.296 INFO:tasks.ceph:[global] osd_pg_log_trim_min = 0 2026-02-20T11:58:10.296 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2026-02-20T11:58:10.296 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2026-02-20T11:58:10.296 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-20T11:58:10.296 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-20T11:58:10.296 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-20T11:58:10.297 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-20T11:58:10.297 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-20T11:58:10.297 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-20T11:58:10.297 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-20T11:58:10.297 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-20T11:58:10.297 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-20T11:58:10.297 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-20T11:58:10.297 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-20T11:58:10.297 INFO:tasks.ceph:Setting up mon.a... 2026-02-20T11:58:10.298 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-20T11:58:10.341 INFO:teuthology.orchestra.run.trial153.stdout:creating /etc/ceph/ceph.keyring 2026-02-20T11:58:10.344 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-20T11:58:10.430 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-20T11:58:10.503 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.153')] 2026-02-20T11:58:10.504 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.153', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-20T11:58:10.505 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:10.505 DEBUG:teuthology.orchestra.run.trial153:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-20T11:58:10.560 DEBUG:teuthology.orchestra.run.trial153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.153 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:58:10.640 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:10.634+0000 7f3ce40a5f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:10.641 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:10.634+0000 7f3ce40a5f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:10.641 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:10.635+0000 7f3ce40a5f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:10.642 INFO:teuthology.orchestra.run.trial153.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:58:10.642 INFO:teuthology.orchestra.run.trial153.stdout:monmaptool: generated fsid 722e2839-9905-4d5d-898b-9fe4f73a126b 2026-02-20T11:58:10.642 INFO:teuthology.orchestra.run.trial153.stdout:setting min_mon_release = tentacle 2026-02-20T11:58:10.642 INFO:teuthology.orchestra.run.trial153.stdout:epoch 0 2026-02-20T11:58:10.642 INFO:teuthology.orchestra.run.trial153.stdout:fsid 722e2839-9905-4d5d-898b-9fe4f73a126b 2026-02-20T11:58:10.642 INFO:teuthology.orchestra.run.trial153.stdout:last_changed 2026-02-20T11:58:10.635722+0000 2026-02-20T11:58:10.643 INFO:teuthology.orchestra.run.trial153.stdout:created 2026-02-20T11:58:10.635722+0000 2026-02-20T11:58:10.643 INFO:teuthology.orchestra.run.trial153.stdout:min_mon_release 20 (tentacle) 2026-02-20T11:58:10.643 INFO:teuthology.orchestra.run.trial153.stdout:election_strategy: 1 2026-02-20T11:58:10.643 INFO:teuthology.orchestra.run.trial153.stdout:0: [v2:10.20.193.153:3300/0,v1:10.20.193.153:6789/0] mon.a 2026-02-20T11:58:10.643 INFO:teuthology.orchestra.run.trial153.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-20T11:58:10.644 DEBUG:teuthology.orchestra.run.trial153:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-20T11:58:10.702 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 722e2839-9905-4d5d-898b-9fe4f73a126b... 2026-02-20T11:58:10.703 DEBUG:teuthology.orchestra.run.trial153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-20T11:58:10.744 DEBUG:teuthology.orchestra.run.trial183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-20T11:58:10.783 INFO:teuthology.orchestra.run.trial153.stdout:[global] 2026-02-20T11:58:10.783 INFO:teuthology.orchestra.run.trial153.stdout: chdir = "" 2026-02-20T11:58:10.783 INFO:teuthology.orchestra.run.trial153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-20T11:58:10.783 INFO:teuthology.orchestra.run.trial153.stdout: auth supported = cephx 2026-02-20T11:58:10.783 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.783 INFO:teuthology.orchestra.run.trial153.stdout: filestore xattr use omap = true 2026-02-20T11:58:10.783 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.783 INFO:teuthology.orchestra.run.trial153.stdout: mon clock drift allowed = 1.000 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: osd crush chooseleaf type = 0 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: auth debug = true 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: ms die on old message = true 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: ms die on bug = true 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: mon pg warn max object skew = 0 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: # disable pg_autoscaler by default for new pools 2026-02-20T11:58:10.784 INFO:teuthology.orchestra.run.trial153.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: osd pool default size = 2 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: mon osd allow primary affinity = true 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: mon osd allow pg remap = true 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: mon warn on legacy crush tunables = false 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: mon warn on crush straw calc version zero = false 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: mon warn on no sortbitwise = false 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: mon warn on osd down out interval zero = false 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: mon warn on too few osds = false 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-20T11:58:10.785 INFO:teuthology.orchestra.run.trial153.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: mon_allow_pool_size_one = true 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: osd default data pool replay window = 5 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: mon allow pool delete = true 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: mon cluster log file level = debug 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: debug asserts on shutdown = true 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: mon health detail to clog = false 2026-02-20T11:58:10.786 INFO:teuthology.orchestra.run.trial153.stdout: mon host = 10.20.193.153 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: ms client mode = crc 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: ms cluster mode = crc 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: ms mon client mode = crc 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: ms mon cluster mode = crc 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: ms mon service mode = crc 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: ms service mode = crc 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: osd_max_pg_log_entries = 2 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: osd_min_pg_log_entries = 1 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: osd_pg_log_trim_min = 0 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: osd_pool_default_min_size = 2 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: osd_pool_default_size = 3 2026-02-20T11:58:10.787 INFO:teuthology.orchestra.run.trial153.stdout: fsid = 722e2839-9905-4d5d-898b-9fe4f73a126b 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout:[osd] 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: osd journal size = 100 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: osd scrub load threshold = 5.0 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: osd scrub max interval = 120 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: osd mclock profile = high_recovery_ops 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: osd mclock skip benchmark = true 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: osd recover clone overlap = true 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: osd recovery max chunk = 1048576 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.788 INFO:teuthology.orchestra.run.trial153.stdout: osd debug shutdown = true 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: osd debug op order = true 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: osd debug verify stray on activate = true 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: osd debug trim objects = true 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: osd open classes on start = true 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: osd debug pg log writeout = true 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: osd deep scrub update digest min age = 30 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: osd map max advance = 10 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: journal zero on create = true 2026-02-20T11:58:10.789 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: filestore ondisk finisher threads = 3 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: filestore apply finisher threads = 3 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: bdev debug aio = true 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: osd debug misdirected ops = true 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: debug monc = 20 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: crimson cpu num = 4 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: crimson osd obc lru size = 10 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: debug ms = 20 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: debug osd = 20 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore = 20 2026-02-20T11:58:10.790 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore backref = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore cache = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore cleaner = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore device = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore epm = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore fixedkv tree = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore journal = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore lba = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore odata = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore omap = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore onode = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore t = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: debug seastore tm = 20 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: osd blocked scrub grace period = 3600 2026-02-20T11:58:10.791 INFO:teuthology.orchestra.run.trial153.stdout: osd debug reject backfill probability = 0.3 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: osd delete sleep = 1 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: osd max backfills = 3 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: osd max markdown count = 1000 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: osd objectstore = seastore 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: osd scrub min interval = 60 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: osd shutdown pgref assert = True 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: osd snap trim sleep = 2 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: osd target transaction size = 5 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: seastore cachepin size pershard = 64M 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: seastore cachepin type = 2Q 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: seastore max concurrent transactions = 8 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.792 INFO:teuthology.orchestra.run.trial153.stdout:[mgr] 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: debug ms = 1 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: debug mgr = 20 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: debug mon = 20 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: debug auth = 20 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: mon reweight min pgs per osd = 4 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: mon reweight min bytes per osd = 10 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: mgr/telemetry/nag = false 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout:[mon] 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: debug ms = 1 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: debug mon = 20 2026-02-20T11:58:10.793 INFO:teuthology.orchestra.run.trial153.stdout: debug paxos = 20 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: debug auth = 20 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: mon data avail warn = 5 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: mon mgr mkfs grace = 240 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: mon reweight min pgs per osd = 4 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: mon osd reporter subtree level = osd 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: mon osd prime pg temp = true 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: mon reweight min bytes per osd = 10 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: auth mon ticket ttl = 660 # 11m 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: auth service ticket ttl = 240 # 4m 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.794 INFO:teuthology.orchestra.run.trial153.stdout: # don't complain about insecure global_id in the test suite 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: # 1m isn't quite enough 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: mon_down_mkfs_grace = 2m 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: mon_warn_on_filestore_osds = false 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: mon min osdmap epochs = 50 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: mon osdmap full prune interval = 2 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: mon osdmap full prune min = 15 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: mon osdmap full prune txsize = 2 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: osd pool default crimson = True 2026-02-20T11:58:10.795 INFO:teuthology.orchestra.run.trial153.stdout: paxos service trim min = 10 2026-02-20T11:58:10.796 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:10.796 INFO:teuthology.orchestra.run.trial153.stdout:[client] 2026-02-20T11:58:10.796 INFO:teuthology.orchestra.run.trial153.stdout: rgw cache enabled = true 2026-02-20T11:58:10.796 INFO:teuthology.orchestra.run.trial153.stdout: rgw enable ops log = true 2026-02-20T11:58:10.796 INFO:teuthology.orchestra.run.trial153.stdout: rgw enable usage log = true 2026-02-20T11:58:10.796 INFO:teuthology.orchestra.run.trial153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-20T11:58:10.796 INFO:teuthology.orchestra.run.trial153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-20T11:58:10.796 INFO:teuthology.orchestra.run.trial153.stdout:[mon.a] 2026-02-20T11:58:10.798 INFO:teuthology.orchestra.run.trial183.stdout:[global] 2026-02-20T11:58:10.798 INFO:teuthology.orchestra.run.trial183.stdout: chdir = "" 2026-02-20T11:58:10.798 INFO:teuthology.orchestra.run.trial183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-20T11:58:10.798 INFO:teuthology.orchestra.run.trial183.stdout: auth supported = cephx 2026-02-20T11:58:10.798 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: filestore xattr use omap = true 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: mon clock drift allowed = 1.000 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: osd crush chooseleaf type = 0 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: auth debug = true 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: ms die on old message = true 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: ms die on bug = true 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: mon pg warn max object skew = 0 2026-02-20T11:58:10.799 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: # disable pg_autoscaler by default for new pools 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: osd pool default size = 2 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: mon osd allow primary affinity = true 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: mon osd allow pg remap = true 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: mon warn on legacy crush tunables = false 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: mon warn on crush straw calc version zero = false 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: mon warn on no sortbitwise = false 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: mon warn on osd down out interval zero = false 2026-02-20T11:58:10.800 INFO:teuthology.orchestra.run.trial183.stdout: mon warn on too few osds = false 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: mon_allow_pool_size_one = true 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: osd default data pool replay window = 5 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: mon allow pool delete = true 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: mon cluster log file level = debug 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: debug asserts on shutdown = true 2026-02-20T11:58:10.801 INFO:teuthology.orchestra.run.trial183.stdout: mon health detail to clog = false 2026-02-20T11:58:10.802 INFO:teuthology.orchestra.run.trial183.stdout: mon host = 10.20.193.153 2026-02-20T11:58:10.802 INFO:teuthology.orchestra.run.trial183.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-20T11:58:10.802 INFO:teuthology.orchestra.run.trial183.stdout: ms client mode = crc 2026-02-20T11:58:10.802 INFO:teuthology.orchestra.run.trial183.stdout: ms cluster mode = crc 2026-02-20T11:58:10.802 INFO:teuthology.orchestra.run.trial183.stdout: ms mon client mode = crc 2026-02-20T11:58:10.802 INFO:teuthology.orchestra.run.trial183.stdout: ms mon cluster mode = crc 2026-02-20T11:58:10.802 INFO:teuthology.orchestra.run.trial183.stdout: ms mon service mode = crc 2026-02-20T11:58:10.802 INFO:teuthology.orchestra.run.trial183.stdout: ms service mode = crc 2026-02-20T11:58:10.802 INFO:teuthology.orchestra.run.trial183.stdout: osd_max_pg_log_entries = 2 2026-02-20T11:58:10.802 INFO:teuthology.orchestra.run.trial183.stdout: osd_min_pg_log_entries = 1 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout: osd_pg_log_trim_min = 0 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout: osd_pool_default_min_size = 2 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout: osd_pool_default_size = 3 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout: fsid = 722e2839-9905-4d5d-898b-9fe4f73a126b 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout:[osd] 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout: osd journal size = 100 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout: osd scrub load threshold = 5.0 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout: osd scrub max interval = 120 2026-02-20T11:58:10.803 INFO:teuthology.orchestra.run.trial183.stdout: osd mclock profile = high_recovery_ops 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: osd mclock skip benchmark = true 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: osd recover clone overlap = true 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: osd recovery max chunk = 1048576 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: osd debug shutdown = true 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: osd debug op order = true 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: osd debug verify stray on activate = true 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: osd debug trim objects = true 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: osd open classes on start = true 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: osd debug pg log writeout = true 2026-02-20T11:58:10.804 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: osd deep scrub update digest min age = 30 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: osd map max advance = 10 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: journal zero on create = true 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: filestore ondisk finisher threads = 3 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: filestore apply finisher threads = 3 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: bdev debug aio = true 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: osd debug misdirected ops = true 2026-02-20T11:58:10.805 INFO:teuthology.orchestra.run.trial183.stdout: debug monc = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: crimson cpu num = 4 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: crimson osd obc lru size = 10 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug ms = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug osd = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore backref = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore cache = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore cleaner = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore device = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore epm = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore fixedkv tree = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore journal = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore lba = 20 2026-02-20T11:58:10.806 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore odata = 20 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore omap = 20 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore onode = 20 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore t = 20 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore tm = 20 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: osd blocked scrub grace period = 3600 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: osd debug reject backfill probability = 0.3 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: osd delete sleep = 1 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: osd max backfills = 3 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: osd max markdown count = 1000 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: osd objectstore = seastore 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: osd scrub min interval = 60 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: osd shutdown pgref assert = True 2026-02-20T11:58:10.807 INFO:teuthology.orchestra.run.trial183.stdout: osd snap trim sleep = 2 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: osd target transaction size = 5 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: seastore cachepin size pershard = 64M 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: seastore cachepin type = 2Q 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: seastore max concurrent transactions = 8 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout:[mgr] 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: debug ms = 1 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: debug mgr = 20 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: debug mon = 20 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: debug auth = 20 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: mon reweight min pgs per osd = 4 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: mon reweight min bytes per osd = 10 2026-02-20T11:58:10.808 INFO:teuthology.orchestra.run.trial183.stdout: mgr/telemetry/nag = false 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout:[mon] 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: debug ms = 1 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: debug mon = 20 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: debug paxos = 20 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: debug auth = 20 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: mon data avail warn = 5 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: mon mgr mkfs grace = 240 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: mon reweight min pgs per osd = 4 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: mon osd reporter subtree level = osd 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: mon osd prime pg temp = true 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: mon reweight min bytes per osd = 10 2026-02-20T11:58:10.809 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: auth mon ticket ttl = 660 # 11m 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: auth service ticket ttl = 240 # 4m 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: # don't complain about insecure global_id in the test suite 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: # 1m isn't quite enough 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: mon_down_mkfs_grace = 2m 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: mon_warn_on_filestore_osds = false 2026-02-20T11:58:10.810 INFO:teuthology.orchestra.run.trial183.stdout: mon min osdmap epochs = 50 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: mon osdmap full prune interval = 2 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: mon osdmap full prune min = 15 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: mon osdmap full prune txsize = 2 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: osd pool default crimson = True 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: paxos service trim min = 10 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout:[client] 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: rgw cache enabled = true 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: rgw enable ops log = true 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: rgw enable usage log = true 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-20T11:58:10.811 INFO:teuthology.orchestra.run.trial183.stdout:[mon.a] 2026-02-20T11:58:10.812 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-20T11:58:10.812 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-20T11:58:10.890 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-20T11:58:10.890 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:10.890 DEBUG:teuthology.orchestra.run.trial153:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-20T11:58:10.907 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:10.907 DEBUG:teuthology.orchestra.run.trial153:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-20T11:58:10.964 INFO:tasks.ceph:Sending monmap to node ubuntu@trial153.front.sepia.ceph.com 2026-02-20T11:58:10.964 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:10.964 DEBUG:teuthology.orchestra.run.trial153:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-20T11:58:10.964 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-20T11:58:11.037 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:11.037 DEBUG:teuthology.orchestra.run.trial153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:58:11.096 INFO:tasks.ceph:Sending monmap to node ubuntu@trial183.front.sepia.ceph.com 2026-02-20T11:58:11.096 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:58:11.096 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-20T11:58:11.096 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-20T11:58:11.131 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:58:11.131 DEBUG:teuthology.orchestra.run.trial183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:58:11.188 INFO:tasks.ceph:Setting up mon nodes... 2026-02-20T11:58:11.188 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-20T11:58:11.188 DEBUG:teuthology.orchestra.run.trial183:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-20T11:58:11.279 INFO:teuthology.orchestra.run.trial183.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-20T11:58:11.282 INFO:tasks.ceph:Setting up mds nodes... 2026-02-20T11:58:11.282 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-20T11:58:11.282 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-20T11:58:11.325 INFO:teuthology.orchestra.run.trial153.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-20T11:58:11.336 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2026-02-20T11:58:11.380 INFO:teuthology.orchestra.run.trial183.stdout:creating /etc/ceph/ceph.client.1.keyring 2026-02-20T11:58:11.392 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-20T11:58:11.392 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial183.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-20T11:58:11.392 DEBUG:teuthology.orchestra.run.trial153:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-20T11:58:11.419 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-20T11:58:11.420 INFO:tasks.ceph:role: osd.0 2026-02-20T11:58:11.420 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial153.front.sepia.ceph.com 2026-02-20T11:58:11.420 DEBUG:teuthology.orchestra.run.trial153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-20T11:58:11.485 INFO:teuthology.orchestra.run.trial153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T11:58:11.485 INFO:teuthology.orchestra.run.trial153.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T11:58:11.485 INFO:teuthology.orchestra.run.trial153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T11:58:11.485 INFO:teuthology.orchestra.run.trial153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T11:58:11.485 INFO:teuthology.orchestra.run.trial153.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T11:58:11.485 INFO:teuthology.orchestra.run.trial153.stdout: = sunit=0 swidth=0 blks 2026-02-20T11:58:11.486 INFO:teuthology.orchestra.run.trial153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T11:58:11.486 INFO:teuthology.orchestra.run.trial153.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T11:58:11.486 INFO:teuthology.orchestra.run.trial153.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T11:58:11.486 INFO:teuthology.orchestra.run.trial153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T11:58:12.060 INFO:teuthology.orchestra.run.trial153.stdout:Discarding blocks...Done. 2026-02-20T11:58:12.063 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial153.front.sepia.ceph.com -o noatime 2026-02-20T11:58:12.063 DEBUG:teuthology.orchestra.run.trial153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-20T11:58:12.133 DEBUG:teuthology.orchestra.run.trial153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-20T11:58:12.200 DEBUG:teuthology.orchestra.run.trial153:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-20T11:58:12.265 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-20T11:58:12.265 INFO:tasks.ceph:role: osd.1 2026-02-20T11:58:12.265 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial153.front.sepia.ceph.com 2026-02-20T11:58:12.266 DEBUG:teuthology.orchestra.run.trial153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-20T11:58:12.329 INFO:teuthology.orchestra.run.trial153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T11:58:12.329 INFO:teuthology.orchestra.run.trial153.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T11:58:12.330 INFO:teuthology.orchestra.run.trial153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T11:58:12.330 INFO:teuthology.orchestra.run.trial153.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T11:58:12.330 INFO:teuthology.orchestra.run.trial153.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T11:58:12.330 INFO:teuthology.orchestra.run.trial153.stdout: = sunit=0 swidth=0 blks 2026-02-20T11:58:12.330 INFO:teuthology.orchestra.run.trial153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T11:58:12.330 INFO:teuthology.orchestra.run.trial153.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T11:58:12.330 INFO:teuthology.orchestra.run.trial153.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T11:58:12.330 INFO:teuthology.orchestra.run.trial153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T11:58:12.902 INFO:teuthology.orchestra.run.trial153.stdout:Discarding blocks...Done. 2026-02-20T11:58:12.904 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial153.front.sepia.ceph.com -o noatime 2026-02-20T11:58:12.904 DEBUG:teuthology.orchestra.run.trial153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-20T11:58:12.972 DEBUG:teuthology.orchestra.run.trial153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-20T11:58:13.039 DEBUG:teuthology.orchestra.run.trial153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:58:13.189 INFO:teuthology.orchestra.run.trial153.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T11:58:13.244 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:13,239 seastar - Reactor backend: linux-aio 2026-02-20T11:58:13.262 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:13,257 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:13.262 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:13,257 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:13.264 INFO:teuthology.orchestra.run.trial153.stderr:==39368==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:58:13.322 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:13,317 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-20T11:58:13.322 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:13,317 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-20T11:58:13.323 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:13,317 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-20T11:58:13.323 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:13,317 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-20T11:58:13.323 INFO:teuthology.orchestra.run.trial153.stderr:DEBUG 2026-02-20 11:58:13,317 [shard 0:main] seastar - reactor::drain 2026-02-20T11:58:15.398 INFO:teuthology.orchestra.run.trial153.stderr:----------------------------------------------------- 2026-02-20T11:58:15.398 INFO:teuthology.orchestra.run.trial153.stderr:Suppressions used: 2026-02-20T11:58:15.398 INFO:teuthology.orchestra.run.trial153.stderr: count bytes template 2026-02-20T11:58:15.398 INFO:teuthology.orchestra.run.trial153.stderr: 1 8 MallocExtension::Register 2026-02-20T11:58:15.399 INFO:teuthology.orchestra.run.trial153.stderr:----------------------------------------------------- 2026-02-20T11:58:15.399 INFO:teuthology.orchestra.run.trial153.stderr: 2026-02-20T11:58:15.418 INFO:teuthology.orchestra.run.trial153.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T11:58:15.418 INFO:teuthology.orchestra.run.trial153.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T11:58:15.457 INFO:teuthology.orchestra.run.trial153.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T11:58:15.457 INFO:teuthology.orchestra.run.trial153.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T11:58:15.463 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:15,458 seastar - Reactor backend: linux-aio 2026-02-20T11:58:15.480 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:15,475 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:15.480 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:15,475 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:15.484 INFO:teuthology.orchestra.run.trial153.stderr:==39366==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:58:15.484 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:15,479 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T11:58:15.543 INFO:teuthology.orchestra.run.trial153.stderr:DEBUG 2026-02-20 11:58:15,538 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-20T11:58:15.625 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xeacf /lib64/libasan.so.8+0xec776 0xcc3105 0x6f45cbe 0x7b9c9b3 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.625 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x9e10b32 0x9eee799 0x9eee806 0x9f17355 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.625 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0xa9a91f 0xca7ec8 0xcb9f6b 0x9e103e7 0x9eda276 0x9f1a470 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.657 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xfcb03 /lib64/libasan.so.8+0xb13ef 0xb7b689 0x10a36ac 0xf4865a 0x9e1e492 0x9e1e78f 0x9e1ea02 0x9e1fa02 0x9e1ff70 0x9e20295 0x9e203ff 0x9f07c75 0x9f1aa78 0x9edc6d1 0x9edb6d1 0x7ba0c22 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f76cf2 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:15.700 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191ccd /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x1367182 0x29df4ae 0x7f6c018 0x7f6c5ce 0x7f6c6fa 0x7f6f26f 0x7f6fec3 0x7f70184 0x7e5a8a9 0x7fb3f68 0x911dcdf 0x912064d 0x7862725 0x7a52d48 0x6f770d1 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.700 INFO:teuthology.orchestra.run.trial153.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffaf15fe1b 0xffffffffae78ab94 0xffffffffae78ac8b 0xffffffffae7cb948 0xffffffffae7d3aeb 0xffffffffae7d3f4b 0xffffffffae4910d2 0xffffffffaf14d451 0xffffffffaf200ba2 2026-02-20T11:58:15.701 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x361f /lib64/libasan.so.8+0x15197 0x31fb0c9 0x9e1e943 0x9e1fa02 0x9e1ff70 0x9e20295 0x9e203ff 0x9f07c75 0x9f1aa78 0x9edc6d1 0x9edb6d1 0x7f67185 0x7fb45b4 0x911dcdf 0x912064d 0x7862725 0x7a52d48 0x6f770d1 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.725 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x297f46e 0x29d89ca 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:15.766 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d83f9 0x7d69b2a 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.766 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x297f469 0x29d89ca 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.767 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 66 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ed7 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.887 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0xccaa51 0x1138f5f 0xf4864f 0x9e1e2c5 0x9e1e78f 0x9e1ea02 0x9e1ecb8 0x9e1f27a 0x9e27721 0x9e27e01 0x9f17e54 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.887 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x9e1ebd5 0x9e1f27a 0x9e27721 0x9e27e01 0x9f17e54 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.887 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15802 0x9e1ea56 0x9e1ecb8 0x9e1f767 0x9e1fc65 0x9e20295 0x9e203ff 0x9f07c75 0x9f1aa78 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.970 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d88e8 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:15.985 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d8872 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.986 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d79b2 0x7d69b08 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:15.986 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d79f6 0x7d69b08 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:18.190 INFO:teuthology.orchestra.run.trial153.stderr:----------------------------------------------------- 2026-02-20T11:58:18.190 INFO:teuthology.orchestra.run.trial153.stderr:Suppressions used: 2026-02-20T11:58:18.190 INFO:teuthology.orchestra.run.trial153.stderr: count bytes template 2026-02-20T11:58:18.190 INFO:teuthology.orchestra.run.trial153.stderr: 1 8 MallocExtension::Register 2026-02-20T11:58:18.190 INFO:teuthology.orchestra.run.trial153.stderr:----------------------------------------------------- 2026-02-20T11:58:18.190 INFO:teuthology.orchestra.run.trial153.stderr: 2026-02-20T11:58:18.227 DEBUG:teuthology.orchestra.run.trial153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-20T11:58:18.253 DEBUG:teuthology.orchestra.run.trial153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:58:18.410 INFO:teuthology.orchestra.run.trial153.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T11:58:18.462 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:18,457 seastar - Reactor backend: linux-aio 2026-02-20T11:58:18.476 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:18,471 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:18.476 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:18,471 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:18.478 INFO:teuthology.orchestra.run.trial153.stderr:==39432==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:58:18.535 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:18,530 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-20T11:58:18.535 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:18,530 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-20T11:58:18.536 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:18,530 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-20T11:58:18.536 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:18,530 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-20T11:58:18.536 INFO:teuthology.orchestra.run.trial153.stderr:DEBUG 2026-02-20 11:58:18,530 [shard 0:main] seastar - reactor::drain 2026-02-20T11:58:20.622 INFO:teuthology.orchestra.run.trial153.stderr:----------------------------------------------------- 2026-02-20T11:58:20.622 INFO:teuthology.orchestra.run.trial153.stderr:Suppressions used: 2026-02-20T11:58:20.622 INFO:teuthology.orchestra.run.trial153.stderr: count bytes template 2026-02-20T11:58:20.622 INFO:teuthology.orchestra.run.trial153.stderr: 1 8 MallocExtension::Register 2026-02-20T11:58:20.622 INFO:teuthology.orchestra.run.trial153.stderr:----------------------------------------------------- 2026-02-20T11:58:20.622 INFO:teuthology.orchestra.run.trial153.stderr: 2026-02-20T11:58:20.643 INFO:teuthology.orchestra.run.trial153.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T11:58:20.643 INFO:teuthology.orchestra.run.trial153.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T11:58:20.688 INFO:teuthology.orchestra.run.trial153.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T11:58:20.688 INFO:teuthology.orchestra.run.trial153.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T11:58:20.694 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:20,690 seastar - Reactor backend: linux-aio 2026-02-20T11:58:20.707 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:20,702 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:20.707 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:20,702 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:20.710 INFO:teuthology.orchestra.run.trial153.stderr:==39430==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:58:20.710 INFO:teuthology.orchestra.run.trial153.stderr:INFO 2026-02-20 11:58:20,706 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T11:58:20.770 INFO:teuthology.orchestra.run.trial153.stderr:DEBUG 2026-02-20 11:58:20,766 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-20T11:58:20.853 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9cf0b5b 0x9d70916 0x9d70a2c 0x9d70b08 0x9d70d84 0x9d70e3d 0x9d7132b 0x9d7172f 0x9d719ec 0x9d72834 0x9ef0163 0x9ef025d 0x7b9c6c3 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:20.853 INFO:teuthology.orchestra.run.trial153.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffaea54da4 0xffffffffae80f59b 0xffffffffae7c95de 0xffffffffae7d3aeb 0xffffffffae7d3f4b 0xffffffffae4910d2 0xffffffffaf14d451 0xffffffffaf200ba2 2026-02-20T11:58:20.853 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe419a1 0x9f1dc43 0x78bcf19 0x7b9c686 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:20.853 INFO:teuthology.orchestra.run.trial153.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffae61f6b4 0xffffffffae61fb20 0xffffffffae49525e 0xffffffffae7e61a9 0xffffffffae7c9715 0xffffffffae7d3aeb 0xffffffffae7d3f4b 0xffffffffae4910d2 0xffffffffaf14d451 0xffffffffaf200ba2 2026-02-20T11:58:20.853 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1294d2 /lib64/libasan.so.8+0x1297ad /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0x10a3723 0xf4865a 0x9e1e2c5 0x9e1e78f 0x9e1ea02 0x9e1ecb8 0x9e1f767 0x9e1fc65 0x9e20295 0x9e203ff 0x9f07c75 0x9f1aa78 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:20.886 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb09ea /lib64/libasan.so.8+0xb1437 0xb7b689 0x10a36ac 0xf4865a 0x9e22cec 0x9e2362b 0x9f16b01 0x9f17a21 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b8bc1a 0x7ba0607 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f76cf2 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:20.929 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0xcc3257 0x1138f42 0xf4864f 0x9e22cec 0x9e2362b 0x9f16b01 0x9f195df 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x911c2c7 0x911d4dc 0x912064d 0x7862725 0x7a52d48 0x6f770d1 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:20.929 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x191cf5 /lib64/libasan.so.8+0xc748 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x1367182 0x29df4ae 0x7f6c018 0x7f6c5ce 0x7f6c6fa 0x7f6f26f 0x7f6fec3 0x7f70184 0x7e5a8a9 0x7fb3f68 0x911dcdf 0x912064d 0x7862725 0x7a52d48 0x6f770d1 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:20.929 INFO:teuthology.orchestra.run.trial153.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffae7c8707 0xffffffffae7d3f88 0xffffffffae4910d2 0xffffffffaf14d451 0xffffffffaf200ba2 2026-02-20T11:58:20.929 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0xcc324f 0x10a3600 0xf4865a 0x9e22cec 0x9e22f99 0x9f17991 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9e526 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f76cf2 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:20.954 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x12985d /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:20.995 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1fb10 /lib64/libasan.so.8+0x15197 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:20.995 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d88a3 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:20.998 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d7df9 0x7d69b1d 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:21.121 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x9eee7a4 0x9eee806 0x9f17355 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:21.121 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c6 /lib64/libasan.so.8+0x15197 0x1138fc8 0xf4864f 0x9e1e2c5 0x9e1e78f 0x9e1ea02 0x9e1ecb8 0x9e1f767 0x9e1fc65 0x9e20295 0x9e203ff 0x9f07c75 0x9f1aa78 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:21.121 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x1138fc8 0xf4864f 0x9e1e492 0x9e1e78f 0x9e1ea02 0x9e1ecb8 0x9e1f27a 0x9e27721 0x9e27e01 0x9f17e54 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:21.187 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d885f 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:21.203 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151b5 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:21.219 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d83f9 0x7d69b2a 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:21.220 INFO:teuthology.orchestra.run.trial153.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1521e 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:23.370 INFO:teuthology.orchestra.run.trial153.stderr:----------------------------------------------------- 2026-02-20T11:58:23.370 INFO:teuthology.orchestra.run.trial153.stderr:Suppressions used: 2026-02-20T11:58:23.371 INFO:teuthology.orchestra.run.trial153.stderr: count bytes template 2026-02-20T11:58:23.371 INFO:teuthology.orchestra.run.trial153.stderr: 1 8 MallocExtension::Register 2026-02-20T11:58:23.371 INFO:teuthology.orchestra.run.trial153.stderr:----------------------------------------------------- 2026-02-20T11:58:23.371 INFO:teuthology.orchestra.run.trial153.stderr: 2026-02-20T11:58:23.405 DEBUG:teuthology.orchestra.run.trial153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-20T11:58:23.430 DEBUG:teuthology.orchestra.run.trial183:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-20T11:58:23.456 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-20T11:58:23.457 INFO:tasks.ceph:role: osd.2 2026-02-20T11:58:23.457 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial183.front.sepia.ceph.com 2026-02-20T11:58:23.457 DEBUG:teuthology.orchestra.run.trial183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-20T11:58:23.523 INFO:teuthology.orchestra.run.trial183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T11:58:23.523 INFO:teuthology.orchestra.run.trial183.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T11:58:23.524 INFO:teuthology.orchestra.run.trial183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T11:58:23.524 INFO:teuthology.orchestra.run.trial183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T11:58:23.524 INFO:teuthology.orchestra.run.trial183.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T11:58:23.524 INFO:teuthology.orchestra.run.trial183.stdout: = sunit=0 swidth=0 blks 2026-02-20T11:58:23.524 INFO:teuthology.orchestra.run.trial183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T11:58:23.524 INFO:teuthology.orchestra.run.trial183.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T11:58:23.524 INFO:teuthology.orchestra.run.trial183.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T11:58:23.524 INFO:teuthology.orchestra.run.trial183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T11:58:24.138 INFO:teuthology.orchestra.run.trial183.stdout:Discarding blocks...Done. 2026-02-20T11:58:24.141 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial183.front.sepia.ceph.com -o noatime 2026-02-20T11:58:24.141 DEBUG:teuthology.orchestra.run.trial183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2026-02-20T11:58:24.209 DEBUG:teuthology.orchestra.run.trial183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-20T11:58:24.278 DEBUG:teuthology.orchestra.run.trial183:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-20T11:58:24.342 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-20T11:58:24.343 INFO:tasks.ceph:role: osd.3 2026-02-20T11:58:24.343 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial183.front.sepia.ceph.com 2026-02-20T11:58:24.343 DEBUG:teuthology.orchestra.run.trial183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-20T11:58:24.408 INFO:teuthology.orchestra.run.trial183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T11:58:24.409 INFO:teuthology.orchestra.run.trial183.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T11:58:24.409 INFO:teuthology.orchestra.run.trial183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T11:58:24.409 INFO:teuthology.orchestra.run.trial183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-20T11:58:24.409 INFO:teuthology.orchestra.run.trial183.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T11:58:24.409 INFO:teuthology.orchestra.run.trial183.stdout: = sunit=0 swidth=0 blks 2026-02-20T11:58:24.409 INFO:teuthology.orchestra.run.trial183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T11:58:24.409 INFO:teuthology.orchestra.run.trial183.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T11:58:24.409 INFO:teuthology.orchestra.run.trial183.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T11:58:24.409 INFO:teuthology.orchestra.run.trial183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T11:58:24.967 INFO:teuthology.orchestra.run.trial183.stdout:Discarding blocks...Done. 2026-02-20T11:58:24.969 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial183.front.sepia.ceph.com -o noatime 2026-02-20T11:58:24.970 DEBUG:teuthology.orchestra.run.trial183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2026-02-20T11:58:25.042 DEBUG:teuthology.orchestra.run.trial183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-20T11:58:25.109 DEBUG:teuthology.orchestra.run.trial183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:58:25.259 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T11:58:25.319 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:25,316 seastar - Reactor backend: linux-aio 2026-02-20T11:58:25.335 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:25,332 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:25.335 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:25,332 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:25.337 INFO:teuthology.orchestra.run.trial183.stderr:==39008==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:58:25.393 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:25,391 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-20T11:58:25.393 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:25,391 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-20T11:58:25.394 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:25,391 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-20T11:58:25.394 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:25,391 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-20T11:58:25.394 INFO:teuthology.orchestra.run.trial183.stderr:DEBUG 2026-02-20 11:58:25,391 [shard 0:main] seastar - reactor::drain 2026-02-20T11:58:27.461 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-20T11:58:27.461 INFO:teuthology.orchestra.run.trial183.stderr:Suppressions used: 2026-02-20T11:58:27.461 INFO:teuthology.orchestra.run.trial183.stderr: count bytes template 2026-02-20T11:58:27.461 INFO:teuthology.orchestra.run.trial183.stderr: 1 8 MallocExtension::Register 2026-02-20T11:58:27.461 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-20T11:58:27.461 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-20T11:58:27.480 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T11:58:27.480 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T11:58:27.521 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T11:58:27.522 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T11:58:27.528 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:27,526 seastar - Reactor backend: linux-aio 2026-02-20T11:58:27.537 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:27,534 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:27.537 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:27,535 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:27.540 INFO:teuthology.orchestra.run.trial183.stderr:==39006==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:58:27.540 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:27,538 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T11:58:27.573 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x3469970 0x3469bdd 0x3469e9b 0x347a124 0x350c757 0x350c956 0x350cabd 0x350cc96 0x350d21f 0x337984d 0x33d6b25 0x45f3998 0x467ed29 0x46801d1 0x4680256 0x4680358 0x4680450 0x468065b 0x4680841 0x46808bc 0x4680937 0x46809b2 0x468131f 0x46816c0 0x46818eb 0x4681b52 0x4681e1d 0x468203b 0x46820f3 0x46821ab 0x4682264 0x468231c 0x46823e2 0x46824c3 0x30ae778 0x30ae87e 0x30ae936 0x30ae9ee 0x30aeaa6 0x30cf505 0x317e2e7 0x317e6a8 0x317e7ea 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:27.599 INFO:teuthology.orchestra.run.trial183.stderr:DEBUG 2026-02-20 11:58:27,597 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-20T11:58:27.682 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc447 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe2fb4f 0xe2fe72 0x9efafa4 0x9f17817 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.683 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1518b 0x1138fc8 0xf4864f 0x9e21fb0 0x9e2225d 0x9f1e6c9 0x78bcf19 0x7b9c081 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.683 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x320079a 0x32008d2 0x9f1aa95 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.715 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151b5 0x31fb0c9 0x9e1e943 0x9e1ecb8 0x9e1f767 0x9e1fc65 0x9e20295 0x9e203ff 0x9f07c75 0x9f1aa78 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f76cf2 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:27.756 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x9f16baf 0x9f1979d 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7f67185 0x7fb45b4 0x911dcdf 0x912064d 0x7862725 0x7a52d48 0x6f770d1 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.758 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0xbcb310 0xd08566 0x1852eee 0x9efabe7 0x9efaf8c 0x9f19778 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7f67185 0x7fb45b4 0x911dcdf 0x912064d 0x7862725 0x7a52d48 0x6f770d1 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.760 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc3af /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe419a1 0x9f1acf4 0x9edc6d1 0x9edb6d1 0x7b8bc1a 0x7ba0473 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f76cf2 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.786 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d8330 0x7d69b2a 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:27.822 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151cc 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.824 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.829 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc6583 /lib64/libasan.so.8+0x15197 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.948 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/ld-linux-x86-64.so.2+0x16ed3 /lib64/libasan.so.8+0xc65b1 /lib64/libasan.so.8+0x15197 0x31fb0c9 0x9e1e943 0x9e1ecb8 0x9e1f27a 0x9e27721 0x9e27e01 0x9f17e54 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.949 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb13a7 0xb7b689 0x10a36ac 0xf4865a 0x9e1e492 0x9e1e78f 0x9e1ea02 0x9e20729 0x9e20c76 0x9e20e75 0x9f0d69d 0x9f0ddf8 0x9f18086 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:27.949 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x129787 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x157c7 0x3549a64 0x3549be4 0x359d2e4 0x9e30705 0x9e3080d 0x9ef7d44 0x9ef7ed9 0x9ef826b 0x9ef9851 0x9efab6c 0x9efaf8c 0x9f17817 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:28.031 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:28.047 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d82e4 0x7d69b2a 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:28.047 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:28.048 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x297f494 0x29d89ca 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:30.203 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-20T11:58:30.204 INFO:teuthology.orchestra.run.trial183.stderr:Suppressions used: 2026-02-20T11:58:30.204 INFO:teuthology.orchestra.run.trial183.stderr: count bytes template 2026-02-20T11:58:30.204 INFO:teuthology.orchestra.run.trial183.stderr: 1 8 MallocExtension::Register 2026-02-20T11:58:30.204 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-20T11:58:30.204 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-20T11:58:30.238 DEBUG:teuthology.orchestra.run.trial183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-20T11:58:30.264 DEBUG:teuthology.orchestra.run.trial183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:58:30.417 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: parsing early config 2026-02-20T11:58:30.478 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:30,476 seastar - Reactor backend: linux-aio 2026-02-20T11:58:30.487 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:30,485 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:30.488 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:30,485 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:30.490 INFO:teuthology.orchestra.run.trial183.stderr:==39072==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:58:30.547 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:30,545 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-20T11:58:30.547 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:30,545 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-20T11:58:30.548 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:30,545 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-20T11:58:30.548 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:30,545 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-20T11:58:30.548 INFO:teuthology.orchestra.run.trial183.stderr:DEBUG 2026-02-20 11:58:30,545 [shard 0:main] seastar - reactor::drain 2026-02-20T11:58:32.614 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-20T11:58:32.614 INFO:teuthology.orchestra.run.trial183.stderr:Suppressions used: 2026-02-20T11:58:32.614 INFO:teuthology.orchestra.run.trial183.stderr: count bytes template 2026-02-20T11:58:32.614 INFO:teuthology.orchestra.run.trial183.stderr: 1 8 MallocExtension::Register 2026-02-20T11:58:32.614 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-20T11:58:32.614 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-20T11:58:32.633 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-20T11:58:32.634 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-20T11:58:32.678 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-20T11:58:32.678 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-20T11:58:32.685 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:32,683 seastar - Reactor backend: linux-aio 2026-02-20T11:58:32.693 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:32,691 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:32.693 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:32,691 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-20T11:58:32.697 INFO:teuthology.orchestra.run.trial183.stderr:==39070==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-20T11:58:32.697 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-20 11:58:32,695 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-20T11:58:32.756 INFO:teuthology.orchestra.run.trial183.stderr:DEBUG 2026-02-20 11:58:32,755 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-20T11:58:32.837 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x15838 0xca7ec8 0xcb9f6b 0x9e6682e 0x9eebb68 0x9eee549 0x9eee75a 0x9eee806 0x9f17355 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:32.837 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x359d10b 0x9e30705 0x9e307c7 0x9ef7d44 0x9ef7ed9 0x9ef826b 0x9ef9851 0x9efab6c 0x9efaf8c 0x9f17817 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:32.837 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9ef014c 0x9ef025d 0x7b9c6c3 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f75f2c 0x6f79bf7 0x6f7f709 0x73d6e81 0x73d6f06 0x73d7008 0x73d7100 0x74a009e 0x7246aa6 0x727c254 0x73c1550 0x7413540 0x7246b2d 0x727c45b 0x756db47 0x73d7185 0x73d7263 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:32.837 INFO:teuthology.orchestra.run.trial183.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff9481f6b4 0xffffffff9481fb20 0xffffffff9469525e 0xffffffff949e61a9 0xffffffff949c9715 0xffffffff949d3aeb 0xffffffff949d3f4b 0xffffffff946910d2 0xffffffff9534d451 0xffffffff95400ba2 2026-02-20T11:58:32.870 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x9dda3fe 0x9f0b43a 0x9f0d258 0x9f0d8b2 0x9f0ddf8 0x9f19c81 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x84f2a76 0x7e493f4 0x7e49602 0x7a3f26a 0x718b8df 0x6f76cf2 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:32.912 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x10a3710 0xf4865a 0x9e1e2c5 0x9e1e78f 0x9e1ea02 0x9e1ecb8 0x9e1f27a 0x9e27721 0x9e27e01 0x9f17e54 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b89491 0x7ba0607 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f76cf2 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:32.913 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x4cb12d6 0x9e2cd17 0x9efa76f 0x9efa937 0x9efaa62 0x9efaf8c 0x9f17817 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7f67185 0x7fb45b4 0x911dcdf 0x912064d 0x7862725 0x7a52d48 0x6f770d1 0x7518264 0x75183b8 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:32.937 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x7d69ada 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:32.978 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x151c1 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:32.978 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d8933 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:32.980 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x297f494 0x29d89ca 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:33.100 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cbf3 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x157c7 0x9ea63a1 0x9ea65cd 0x9f1d71b 0x9f1d868 0x9f1d8fb 0x9f1e740 0x78bcf19 0x7b9c081 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:33.100 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x9f088e6 0x9f08bc2 0x9f08f45 0x9f1909c 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:33.100 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libstdc++.so.6+0xc6247 0x31f9a39 0x9e1e6ff 0x9e1ea02 0x9e20729 0x9f0db89 0x9f0ddf8 0x9f18086 0x9f1c4ff 0x9edc6d1 0x9edb6d1 0x7b9c74f 0x7ba6272 0x79ff8c1 0x7a3f1fd 0x718b8df 0x6f2e256 0x6f30572 0x73d4684 0x73d4709 0x73d480b 0x73d4903 0x749ff22 0x7246045 0x727559c 0x73bfa6b 0x740df81 0x72460cc 0x72757a3 0x756b847 0x73d4988 0x73d4a66 0x73bfbe6 0x74a22b8 0x754287a 0x75428f5 0x756b8c8 0x74a0618 0x75d60f4 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:33.183 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x15197 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa145278 0x9dc2113 0x9dc294e 0xefe35b /lib64/libc.so.6+0x2a60f /lib64/libc.so.6+0x2a6bf 0xb58eb4 2026-02-20T11:58:33.198 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d7a4b 0x7d69b08 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:33.198 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f 0x29d838e 0x7d69b2a 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:33.198 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 65 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0xa009912 0xa009bfb 0xa053fd5 0xa0547c8 0xa054a36 0xa054b7e 0xa05503d /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1523c 0x29d8a29 0x7d69aef 0x7e5b15a 0x7e5b9c1 0x7e59663 0x78572bb 0x792ffbe 0x794043b 0x794059a 0x7973fa7 0x797959c 0x7979812 0x7979a12 0x7979a8d 0x7979b08 0x7979d54 0x7acf39f 0x7965b04 0x7aa3401 0x73acf1e 0x73b7ed5 0x747550c 0x760f6dc 0xa0a9e81 0xa006b00 0xa0dacb3 0xa143a5a 0xa15e4b8 0xa15eb34 0x277dec1 0xa780984 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-20T11:58:35.380 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-20T11:58:35.380 INFO:teuthology.orchestra.run.trial183.stderr:Suppressions used: 2026-02-20T11:58:35.380 INFO:teuthology.orchestra.run.trial183.stderr: count bytes template 2026-02-20T11:58:35.380 INFO:teuthology.orchestra.run.trial183.stderr: 1 8 MallocExtension::Register 2026-02-20T11:58:35.380 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-20T11:58:35.380 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-20T11:58:35.415 DEBUG:teuthology.orchestra.run.trial183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-20T11:58:35.442 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-20T11:58:35.442 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:35.442 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-20T11:58:35.469 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:35.469 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-20T11:58:35.535 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:58:35.535 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-20T11:58:35.561 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:58:35.561 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-20T11:58:35.625 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:58:35.625 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-20T11:58:35.689 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:35.689 DEBUG:teuthology.orchestra.run.trial153:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-20T11:58:35.707 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:58:35.707 DEBUG:teuthology.orchestra.run.trial183:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2026-02-20T11:58:35.745 INFO:tasks.ceph:Adding keys to all mons... 2026-02-20T11:58:35.746 DEBUG:teuthology.orchestra.run.trial153:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-20T11:58:35.771 INFO:teuthology.orchestra.run.trial153.stdout:[osd.0] 2026-02-20T11:58:35.771 INFO:teuthology.orchestra.run.trial153.stdout: key = AQBXTJhpXkpaIBAAPOF48bjHfLtJF4fXdRejTQ== 2026-02-20T11:58:35.771 INFO:teuthology.orchestra.run.trial153.stdout:[osd.1] 2026-02-20T11:58:35.771 INFO:teuthology.orchestra.run.trial153.stdout: key = AQBcTJhpkgvjLRAA04fA5vbhn0oC3GrQeVSR7A== 2026-02-20T11:58:35.771 INFO:teuthology.orchestra.run.trial153.stdout:[mgr.x] 2026-02-20T11:58:35.771 INFO:teuthology.orchestra.run.trial153.stdout: key = AQBTTJhpKraAEBAAzDvPdpf84Cvnw+frNTCy/w== 2026-02-20T11:58:35.771 INFO:teuthology.orchestra.run.trial153.stdout:[osd.2] 2026-02-20T11:58:35.771 INFO:teuthology.orchestra.run.trial153.stdout: key = AQBjTJhptqLUIxAAXyX7T3r6+Nylwj1EP+t89A== 2026-02-20T11:58:35.771 INFO:teuthology.orchestra.run.trial153.stdout:[osd.3] 2026-02-20T11:58:35.772 INFO:teuthology.orchestra.run.trial153.stdout: key = AQBoTJhpLTU4LRAA7Fxc+Q0P/3fobd7ss95oMg== 2026-02-20T11:58:35.772 INFO:teuthology.orchestra.run.trial153.stdout:[client.0] 2026-02-20T11:58:35.772 INFO:teuthology.orchestra.run.trial153.stdout: key = AQBTTJhpbq0XExAAUG4eqNbwUgfWoh5yRRpRjQ== 2026-02-20T11:58:35.772 INFO:teuthology.orchestra.run.trial153.stdout:[client.1] 2026-02-20T11:58:35.772 INFO:teuthology.orchestra.run.trial153.stdout: key = AQBTTJhpP8yLFhAA30DLRboS/R9lmtTAb+emuw== 2026-02-20T11:58:35.773 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T11:58:35.862 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T11:58:35.911 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-20T11:58:35.960 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T11:58:36.009 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T11:58:36.057 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T11:58:36.109 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T11:58:36.161 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-20T11:58:36.162 DEBUG:teuthology.orchestra.run.trial153:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-20T11:58:36.187 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-20T11:58:36.276 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:36.270+0000 7fa7b4a3ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.276 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:36.270+0000 7fa7b4a3ac00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.287 DEBUG:teuthology.orchestra.run.trial153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-20T11:58:36.313 DEBUG:teuthology.orchestra.run.trial153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-20T11:58:36.368 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-20T11:58:36.369 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-20T11:58:36.369 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-20T11:58:36.411 INFO:tasks.ceph.mon.a:Started 2026-02-20T11:58:36.411 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-20T11:58:36.411 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-20T11:58:36.412 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-20T11:58:36.414 INFO:tasks.ceph.mgr.x:Started 2026-02-20T11:58:36.415 DEBUG:tasks.ceph:set 0 configs 2026-02-20T11:58:36.415 DEBUG:teuthology.orchestra.run.trial153:> sudo ceph --cluster ceph config dump 2026-02-20T11:58:36.470 INFO:tasks.ceph.mon.a.trial153.stderr:2026-02-20T11:58:36.464+0000 7f59a393dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.471 INFO:tasks.ceph.mon.a.trial153.stderr:2026-02-20T11:58:36.464+0000 7f59a393dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.471 INFO:tasks.ceph.mon.a.trial153.stderr:2026-02-20T11:58:36.464+0000 7f59a393dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.476 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-20T11:58:36.472+0000 7f4969e87fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.489 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:36.480+0000 7f247da5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.492 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:36.486+0000 7f247da5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.493 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-20T11:58:36.490+0000 7f4969e87fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.493 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-20T11:58:36.490+0000 7f4969e87fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.526 INFO:teuthology.orchestra.run.trial153.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-20T11:58:36.535 INFO:tasks.ceph:Setting crush tunables to default 2026-02-20T11:58:36.535 DEBUG:teuthology.orchestra.run.trial153:> sudo ceph --cluster ceph osd crush tunables default 2026-02-20T11:58:36.622 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:36.613+0000 7fa8eb1b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.625 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:36.619+0000 7fa8eb1b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.659 INFO:teuthology.orchestra.run.trial153.stderr:adjusted tunables profile to default 2026-02-20T11:58:36.667 INFO:tasks.ceph:check_enable_crimson: True 2026-02-20T11:58:36.667 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-20T11:58:36.668 DEBUG:teuthology.orchestra.run.trial153:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-20T11:58:36.741 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:36.732+0000 7fe22ad0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.743 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:36.738+0000 7fe22ad0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:36.784 INFO:tasks.ceph.mon.a.trial153.stderr:2026-02-20T11:58:36.778+0000 7f599bbf1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-20T11:58:36.784 INFO:tasks.ceph.mon.a.trial153.stderr:Please be aware that this feature is experimental, untested, 2026-02-20T11:58:36.784 INFO:tasks.ceph.mon.a.trial153.stderr:unsupported, and may result in data corruption, data loss, 2026-02-20T11:58:36.784 INFO:tasks.ceph.mon.a.trial153.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-20T11:58:36.784 INFO:tasks.ceph.mon.a.trial153.stderr:feature with important data. 2026-02-20T11:58:36.784 INFO:tasks.ceph.mon.a.trial153.stderr: 2026-02-20T11:58:37.345 INFO:tasks.ceph.mgr.x.trial183.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-20T11:58:37.345 INFO:tasks.ceph.mgr.x.trial183.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-20T11:58:37.345 INFO:tasks.ceph.mgr.x.trial183.stderr: from numpy import show_config as show_numpy_config 2026-02-20T11:58:37.671 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-20T11:58:37.672 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:37.672 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-20T11:58:37.696 DEBUG:teuthology.orchestra.run.trial153:> set -ex 2026-02-20T11:58:37.696 DEBUG:teuthology.orchestra.run.trial153:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-20T11:58:37.759 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:58:37.759 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-20T11:58:37.784 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-20T11:58:37.784 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-20T11:58:37.847 DEBUG:teuthology.orchestra.run.trial183:> sudo ceph --cluster ceph osd new 27b61203-7e80-4ea1-9e43-38813bf3565e 0 2026-02-20T11:58:37.960 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-20T11:58:37.954+0000 7f69da2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:37.962 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-20T11:58:37.959+0000 7f69da2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:38.000 INFO:teuthology.orchestra.run.trial183.stdout:0 2026-02-20T11:58:38.008 DEBUG:teuthology.orchestra.run.trial183:> sudo ceph --cluster ceph osd new ba0e97c8-b1b8-4469-b6fb-0dde4f3d6b24 1 2026-02-20T11:58:38.082 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-20T11:58:38.076+0000 7f5f8090b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:38.084 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-20T11:58:38.081+0000 7f5f8090b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:38.122 INFO:teuthology.orchestra.run.trial183.stdout:1 2026-02-20T11:58:38.131 DEBUG:teuthology.orchestra.run.trial183:> sudo ceph --cluster ceph osd new c0b662d1-4522-4fc6-ad46-8c32656984cf 2 2026-02-20T11:58:38.207 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-20T11:58:38.201+0000 7f575be5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:38.210 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-20T11:58:38.207+0000 7f575be5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:38.248 INFO:teuthology.orchestra.run.trial183.stdout:2 2026-02-20T11:58:38.256 DEBUG:teuthology.orchestra.run.trial183:> sudo ceph --cluster ceph osd new acdba941-b115-46e2-ac92-9ddb4e52bd8e 3 2026-02-20T11:58:38.331 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-20T11:58:38.325+0000 7f7a0a41a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:38.334 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-20T11:58:38.331+0000 7f7a0a41a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:38.373 INFO:teuthology.orchestra.run.trial183.stdout:3 2026-02-20T11:58:38.381 DEBUG:teuthology.orchestra.run.trial153:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-20T11:58:38.411 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-20T11:58:38.412 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-20T11:58:38.453 INFO:tasks.ceph.osd.0:Started 2026-02-20T11:58:38.453 DEBUG:teuthology.orchestra.run.trial153:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-20T11:58:38.481 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-20T11:58:38.481 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-20T11:58:38.524 INFO:tasks.ceph.osd.1:Started 2026-02-20T11:58:38.524 DEBUG:teuthology.orchestra.run.trial183:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-20T11:58:38.552 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-20T11:58:38.553 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-20T11:58:38.595 INFO:tasks.ceph.osd.2:Started 2026-02-20T11:58:38.595 DEBUG:teuthology.orchestra.run.trial183:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2026-02-20T11:58:38.625 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-20T11:58:38.625 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2026-02-20T11:58:38.667 INFO:tasks.ceph.osd.3:Started 2026-02-20T11:58:38.667 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T11:58:38.747 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:38.739+0000 7fcefcecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:38.750 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:38.744+0000 7fcefcecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:38.783 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:38.783 INFO:teuthology.orchestra.run.trial153.stdout:{"epoch":7,"fsid":"722e2839-9905-4d5d-898b-9fe4f73a126b","created":"2026-02-20T11:58:36.478281+0000","modified":"2026-02-20T11:58:38.368484+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T11:58:38.368484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"27b61203-7e80-4ea1-9e43-38813bf3565e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ba0e97c8-b1b8-4469-b6fb-0dde4f3d6b24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c0b662d1-4522-4fc6-ad46-8c32656984cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"acdba941-b115-46e2-ac92-9ddb4e52bd8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:58:38.791 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-20T11:58:38.792 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-20T11:58:39.195 DEBUG:teuthology.orchestra.run.trial153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-20T11:58:39.262 INFO:teuthology.misc.health.trial153.stderr:2026-02-20T11:58:39.254+0000 7f4f44ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:39.264 INFO:teuthology.misc.health.trial153.stderr:2026-02-20T11:58:39.259+0000 7f4f44ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:39.297 INFO:teuthology.misc.health.trial153.stdout: 2026-02-20T11:58:39.297 INFO:teuthology.misc.health.trial153.stdout:{"epoch":7,"fsid":"722e2839-9905-4d5d-898b-9fe4f73a126b","created":"2026-02-20T11:58:36.478281+0000","modified":"2026-02-20T11:58:38.368484+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T11:58:38.368484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"27b61203-7e80-4ea1-9e43-38813bf3565e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ba0e97c8-b1b8-4469-b6fb-0dde4f3d6b24","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c0b662d1-4522-4fc6-ad46-8c32656984cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"acdba941-b115-46e2-ac92-9ddb4e52bd8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:58:39.304 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-20T11:58:39.725 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-20T11:58:39.721+0000 7f488536a640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-20T11:58:45.707 DEBUG:teuthology.orchestra.run.trial153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-20T11:58:45.779 INFO:teuthology.misc.health.trial153.stderr:2026-02-20T11:58:45.772+0000 7f8a36b7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:45.781 INFO:teuthology.misc.health.trial153.stderr:2026-02-20T11:58:45.777+0000 7f8a36b7e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:45.894 INFO:teuthology.misc.health.trial153.stdout: 2026-02-20T11:58:45.894 INFO:teuthology.misc.health.trial153.stdout:{"epoch":10,"fsid":"722e2839-9905-4d5d-898b-9fe4f73a126b","created":"2026-02-20T11:58:36.478281+0000","modified":"2026-02-20T11:58:45.054779+0000","last_up_change":"2026-02-20T11:58:45.054779+0000","last_in_change":"2026-02-20T11:58:38.368484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"27b61203-7e80-4ea1-9e43-38813bf3565e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ba0e97c8-b1b8-4469-b6fb-0dde4f3d6b24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6810","nonce":3168964670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6809","nonce":3168964670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6808","nonce":3168964670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6807","nonce":3168964670}]},"public_addr":"10.20.193.153:6810/3168964670","cluster_addr":"10.20.193.153:6809/3168964670","heartbeat_back_addr":"10.20.193.153:6808/3168964670","heartbeat_front_addr":"10.20.193.153:6807/3168964670","state":["exists","up"]},{"osd":2,"uuid":"c0b662d1-4522-4fc6-ad46-8c32656984cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6805","nonce":2477092242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6804","nonce":2477092242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6807","nonce":2477092242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6808","nonce":2477092242}]},"public_addr":"10.20.193.183:6805/2477092242","cluster_addr":"10.20.193.183:6804/2477092242","heartbeat_back_addr":"10.20.193.183:6807/2477092242","heartbeat_front_addr":"10.20.193.183:6808/2477092242","state":["exists","up"]},{"osd":3,"uuid":"acdba941-b115-46e2-ac92-9ddb4e52bd8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:58:45.903 DEBUG:teuthology.misc:2 of 4 OSDs are up 2026-02-20T11:58:52.307 DEBUG:teuthology.orchestra.run.trial153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-20T11:58:52.378 INFO:teuthology.misc.health.trial153.stderr:2026-02-20T11:58:52.371+0000 7f84023fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:52.380 INFO:teuthology.misc.health.trial153.stderr:2026-02-20T11:58:52.376+0000 7f84023fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:52.492 INFO:teuthology.misc.health.trial153.stdout: 2026-02-20T11:58:52.492 INFO:teuthology.misc.health.trial153.stdout:{"epoch":14,"fsid":"722e2839-9905-4d5d-898b-9fe4f73a126b","created":"2026-02-20T11:58:36.478281+0000","modified":"2026-02-20T11:58:49.073768+0000","last_up_change":"2026-02-20T11:58:46.059875+0000","last_in_change":"2026-02-20T11:58:38.368484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T11:58:46.735834+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b61203-7e80-4ea1-9e43-38813bf3565e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6801","nonce":4091145385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6800","nonce":4091145385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6816","nonce":4091145385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6817","nonce":4091145385}]},"public_addr":"10.20.193.153:6801/4091145385","cluster_addr":"10.20.193.153:6800/4091145385","heartbeat_back_addr":"10.20.193.153:6816/4091145385","heartbeat_front_addr":"10.20.193.153:6817/4091145385","state":["exists","up"]},{"osd":1,"uuid":"ba0e97c8-b1b8-4469-b6fb-0dde4f3d6b24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6810","nonce":3168964670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6809","nonce":3168964670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6808","nonce":3168964670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6807","nonce":3168964670}]},"public_addr":"10.20.193.153:6810/3168964670","cluster_addr":"10.20.193.153:6809/3168964670","heartbeat_back_addr":"10.20.193.153:6808/3168964670","heartbeat_front_addr":"10.20.193.153:6807/3168964670","state":["exists","up"]},{"osd":2,"uuid":"c0b662d1-4522-4fc6-ad46-8c32656984cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6805","nonce":2477092242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6804","nonce":2477092242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6807","nonce":2477092242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6808","nonce":2477092242}]},"public_addr":"10.20.193.183:6805/2477092242","cluster_addr":"10.20.193.183:6804/2477092242","heartbeat_back_addr":"10.20.193.183:6807/2477092242","heartbeat_front_addr":"10.20.193.183:6808/2477092242","state":["exists","up"]},{"osd":3,"uuid":"acdba941-b115-46e2-ac92-9ddb4e52bd8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6806","nonce":527839342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6803","nonce":527839342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6812","nonce":527839342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6811","nonce":527839342}]},"public_addr":"10.20.193.183:6806/527839342","cluster_addr":"10.20.193.183:6803/527839342","heartbeat_back_addr":"10.20.193.183:6812/527839342","heartbeat_front_addr":"10.20.193.183:6811/527839342","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:58:52.499 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-02-20T11:58:52.500 INFO:tasks.ceph:Creating RBD pool 2026-02-20T11:58:52.500 DEBUG:teuthology.orchestra.run.trial153:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-20T11:58:52.573 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:52.565+0000 7f91c49ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:52.576 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:52.572+0000 7f91c49ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:53.547 INFO:teuthology.orchestra.run.trial153.stderr:pool 'rbd' created 2026-02-20T11:58:53.559 DEBUG:teuthology.orchestra.run.trial153:> rbd --cluster ceph pool init rbd 2026-02-20T11:58:53.600 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:53.595+0000 7f61a8027240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:53.600 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:53.595+0000 7f61a8027240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:53.603 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:53.598+0000 7f61a8027240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:56.584 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-20T11:58:56.585 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-20T11:58:56.586 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-20T11:58:56.701 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:56.693+0000 7efcad25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:56.703 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:56.698+0000 7efcad25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:56.815 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:56.824 INFO:teuthology.orchestra.run.trial153.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-20T11:58:56.825 INFO:tasks.ceph_manager:config epoch is 1 2026-02-20T11:58:56.825 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-20T11:58:56.825 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-20T11:58:56.825 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-20T11:58:56.900 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:56.892+0000 7fab6875b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:56.903 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:56.898+0000 7fab6875b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.045 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:57.054 INFO:teuthology.orchestra.run.trial153.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6800","nonce":2321443819},{"type":"v1","addr":"10.20.193.183:6801","nonce":2321443819}]},"active_addr":"10.20.193.183:6801/2321443819","active_change":"2026-02-20T11:58:38.714049+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.183:0","nonce":453898737}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.183:0","nonce":3946979952}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.183:0","nonce":3356123904}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.183:0","nonce":2309210831}]}]} 2026-02-20T11:58:57.057 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-20T11:58:57.057 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-20T11:58:57.057 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T11:58:57.133 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.126+0000 7f6c73c11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.135 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.130+0000 7f6c73c11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.244 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:57.244 INFO:teuthology.orchestra.run.trial153.stdout:{"epoch":18,"fsid":"722e2839-9905-4d5d-898b-9fe4f73a126b","created":"2026-02-20T11:58:36.478281+0000","modified":"2026-02-20T11:58:56.551265+0000","last_up_change":"2026-02-20T11:58:46.059875+0000","last_in_change":"2026-02-20T11:58:38.368484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T11:58:46.735834+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-20T11:58:52.690556+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b61203-7e80-4ea1-9e43-38813bf3565e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6801","nonce":4091145385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6800","nonce":4091145385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6816","nonce":4091145385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6817","nonce":4091145385}]},"public_addr":"10.20.193.153:6801/4091145385","cluster_addr":"10.20.193.153:6800/4091145385","heartbeat_back_addr":"10.20.193.153:6816/4091145385","heartbeat_front_addr":"10.20.193.153:6817/4091145385","state":["exists","up"]},{"osd":1,"uuid":"ba0e97c8-b1b8-4469-b6fb-0dde4f3d6b24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6810","nonce":3168964670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6809","nonce":3168964670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6808","nonce":3168964670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6807","nonce":3168964670}]},"public_addr":"10.20.193.153:6810/3168964670","cluster_addr":"10.20.193.153:6809/3168964670","heartbeat_back_addr":"10.20.193.153:6808/3168964670","heartbeat_front_addr":"10.20.193.153:6807/3168964670","state":["exists","up"]},{"osd":2,"uuid":"c0b662d1-4522-4fc6-ad46-8c32656984cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6805","nonce":2477092242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6804","nonce":2477092242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6807","nonce":2477092242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6808","nonce":2477092242}]},"public_addr":"10.20.193.183:6805/2477092242","cluster_addr":"10.20.193.183:6804/2477092242","heartbeat_back_addr":"10.20.193.183:6807/2477092242","heartbeat_front_addr":"10.20.193.183:6808/2477092242","state":["exists","up"]},{"osd":3,"uuid":"acdba941-b115-46e2-ac92-9ddb4e52bd8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6806","nonce":527839342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6803","nonce":527839342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6812","nonce":527839342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6811","nonce":527839342}]},"public_addr":"10.20.193.183:6806/527839342","cluster_addr":"10.20.193.183:6803/527839342","heartbeat_back_addr":"10.20.193.183:6812/527839342","heartbeat_front_addr":"10.20.193.183:6811/527839342","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:58:57.254 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-20T11:58:57.254 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T11:58:57.330 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.322+0000 7f8cd4c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.332 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.327+0000 7f8cd4c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.445 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:58:57.446 INFO:teuthology.orchestra.run.trial153.stdout:{"epoch":18,"fsid":"722e2839-9905-4d5d-898b-9fe4f73a126b","created":"2026-02-20T11:58:36.478281+0000","modified":"2026-02-20T11:58:56.551265+0000","last_up_change":"2026-02-20T11:58:46.059875+0000","last_in_change":"2026-02-20T11:58:38.368484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T11:58:46.735834+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-20T11:58:52.690556+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b61203-7e80-4ea1-9e43-38813bf3565e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6801","nonce":4091145385}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6800","nonce":4091145385}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6816","nonce":4091145385}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6817","nonce":4091145385}]},"public_addr":"10.20.193.153:6801/4091145385","cluster_addr":"10.20.193.153:6800/4091145385","heartbeat_back_addr":"10.20.193.153:6816/4091145385","heartbeat_front_addr":"10.20.193.153:6817/4091145385","state":["exists","up"]},{"osd":1,"uuid":"ba0e97c8-b1b8-4469-b6fb-0dde4f3d6b24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6810","nonce":3168964670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6809","nonce":3168964670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6808","nonce":3168964670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.153:6807","nonce":3168964670}]},"public_addr":"10.20.193.153:6810/3168964670","cluster_addr":"10.20.193.153:6809/3168964670","heartbeat_back_addr":"10.20.193.153:6808/3168964670","heartbeat_front_addr":"10.20.193.153:6807/3168964670","state":["exists","up"]},{"osd":2,"uuid":"c0b662d1-4522-4fc6-ad46-8c32656984cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6805","nonce":2477092242}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6804","nonce":2477092242}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6807","nonce":2477092242}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6808","nonce":2477092242}]},"public_addr":"10.20.193.183:6805/2477092242","cluster_addr":"10.20.193.183:6804/2477092242","heartbeat_back_addr":"10.20.193.183:6807/2477092242","heartbeat_front_addr":"10.20.193.183:6808/2477092242","state":["exists","up"]},{"osd":3,"uuid":"acdba941-b115-46e2-ac92-9ddb4e52bd8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6806","nonce":527839342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6803","nonce":527839342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6812","nonce":527839342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6811","nonce":527839342}]},"public_addr":"10.20.193.183:6806/527839342","cluster_addr":"10.20.193.183:6803/527839342","heartbeat_back_addr":"10.20.193.183:6812/527839342","heartbeat_front_addr":"10.20.193.183:6811/527839342","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T11:58:57.456 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-20T11:58:57.457 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-20T11:58:57.458 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-20T11:58:57.458 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-20T11:58:57.542 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.534+0000 7ff54a1a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.543 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.534+0000 7fbf7e1a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.543 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.534+0000 7f25c8ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.543 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.535+0000 7f63a41ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.544 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.539+0000 7fbf7e1a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.545 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.540+0000 7f63a41ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.545 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.540+0000 7ff54a1a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.545 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.540+0000 7f25c8ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.564 INFO:teuthology.orchestra.run.trial153.stdout:42949672972 2026-02-20T11:58:57.565 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-20T11:58:57.566 INFO:teuthology.orchestra.run.trial153.stdout:47244640267 2026-02-20T11:58:57.567 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-20T11:58:57.568 INFO:teuthology.orchestra.run.trial153.stdout:47244640267 2026-02-20T11:58:57.568 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-20T11:58:57.569 INFO:teuthology.orchestra.run.trial153.stdout:42949672972 2026-02-20T11:58:57.569 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-20T11:58:57.689 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.681+0000 7fa7e4ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.689 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.681+0000 7f5bf7f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.690 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.681+0000 7fb4890cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.690 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.681+0000 7f722490b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.691 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.687+0000 7f5bf7f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.692 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.687+0000 7fb4890cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.692 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.687+0000 7fa7e4ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.692 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:57.687+0000 7f722490b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:57.804 INFO:teuthology.orchestra.run.trial153.stdout:47244640265 2026-02-20T11:58:57.810 INFO:teuthology.orchestra.run.trial153.stdout:42949672970 2026-02-20T11:58:57.813 INFO:teuthology.orchestra.run.trial153.stdout:47244640266 2026-02-20T11:58:57.815 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640265 for osd.3 2026-02-20T11:58:57.816 INFO:teuthology.orchestra.run.trial153.stdout:42949672969 2026-02-20T11:58:57.821 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672972 got 42949672970 for osd.1 2026-02-20T11:58:57.825 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640266 for osd.0 2026-02-20T11:58:57.827 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672972 got 42949672969 for osd.2 2026-02-20T11:58:58.816 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-20T11:58:58.822 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-20T11:58:58.827 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-20T11:58:58.828 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-20T11:58:58.897 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:58.889+0000 7f557041a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:58.899 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:58.895+0000 7f557041a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:58.939 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:58.931+0000 7f7609b62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:58.941 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:58.933+0000 7f8c49d83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:58.942 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:58.937+0000 7f7609b62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:58.944 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:58.939+0000 7f8c49d83640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:58.944 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:58.936+0000 7fd358b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:58.949 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:58:58.944+0000 7fd358b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:58:59.015 INFO:teuthology.orchestra.run.trial153.stdout:47244640265 2026-02-20T11:58:59.026 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640265 for osd.3 2026-02-20T11:58:59.056 INFO:teuthology.orchestra.run.trial153.stdout:42949672970 2026-02-20T11:58:59.061 INFO:teuthology.orchestra.run.trial153.stdout:42949672969 2026-02-20T11:58:59.062 INFO:teuthology.orchestra.run.trial153.stdout:47244640266 2026-02-20T11:58:59.066 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672972 got 42949672970 for osd.1 2026-02-20T11:58:59.072 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672972 got 42949672969 for osd.2 2026-02-20T11:58:59.073 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640266 for osd.0 2026-02-20T11:59:00.027 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-20T11:59:00.067 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-20T11:59:00.073 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-20T11:59:00.074 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-20T11:59:00.107 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.098+0000 7f6eb501a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.110 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.104+0000 7f6eb501a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.149 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.140+0000 7f38dfc35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.151 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.146+0000 7f38dfc35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.185 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.178+0000 7f4a10cb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.186 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.177+0000 7f12002c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.188 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.183+0000 7f12002c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.188 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.183+0000 7f4a10cb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.224 INFO:teuthology.orchestra.run.trial153.stdout:47244640267 2026-02-20T11:59:00.235 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640267 for osd.3 2026-02-20T11:59:00.235 DEBUG:teuthology.parallel:result is None 2026-02-20T11:59:00.267 INFO:teuthology.orchestra.run.trial153.stdout:42949672972 2026-02-20T11:59:00.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672972 got 42949672972 for osd.1 2026-02-20T11:59:00.278 DEBUG:teuthology.parallel:result is None 2026-02-20T11:59:00.301 INFO:teuthology.orchestra.run.trial153.stdout:42949672972 2026-02-20T11:59:00.308 INFO:teuthology.orchestra.run.trial153.stdout:47244640267 2026-02-20T11:59:00.312 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672972 got 42949672972 for osd.2 2026-02-20T11:59:00.312 DEBUG:teuthology.parallel:result is None 2026-02-20T11:59:00.319 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640267 got 47244640267 for osd.0 2026-02-20T11:59:00.319 DEBUG:teuthology.parallel:result is None 2026-02-20T11:59:00.319 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-20T11:59:00.320 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:59:00.437 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.429+0000 7fa13110b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.439 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.434+0000 7fa13110b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.549 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:59:00.550 INFO:teuthology.orchestra.run.trial153.stderr:dumped all 2026-02-20T11:59:00.558 INFO:teuthology.orchestra.run.trial153.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-20T11:58:58.721136+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4224,"kb_used_data":4224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662400,"statfs":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.175724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565536+0000","last_change":"2026-02-20T11:58:56.567811+0000","last_active":"2026-02-20T11:58:56.565536+0000","last_peered":"2026-02-20T11:58:56.565536+0000","last_clean":"2026-02-20T11:58:56.565536+0000","last_became_active":"2026-02-20T11:58:54.614068+0000","last_became_peered":"2026-02-20T11:58:54.614068+0000","last_unstale":"2026-02-20T11:58:56.565536+0000","last_undegraded":"2026-02-20T11:58:56.565536+0000","last_fullsized":"2026-02-20T11:58:56.565536+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566579+0000","last_change":"2026-02-20T11:58:56.568949+0000","last_active":"2026-02-20T11:58:56.566579+0000","last_peered":"2026-02-20T11:58:56.566579+0000","last_clean":"2026-02-20T11:58:56.566579+0000","last_became_active":"2026-02-20T11:58:54.590150+0000","last_became_peered":"2026-02-20T11:58:54.590150+0000","last_unstale":"2026-02-20T11:58:56.566579+0000","last_undegraded":"2026-02-20T11:58:56.566579+0000","last_fullsized":"2026-02-20T11:58:56.566579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565278+0000","last_change":"2026-02-20T11:58:56.567633+0000","last_active":"2026-02-20T11:58:56.565278+0000","last_peered":"2026-02-20T11:58:56.565278+0000","last_clean":"2026-02-20T11:58:56.565278+0000","last_became_active":"2026-02-20T11:58:54.600464+0000","last_became_peered":"2026-02-20T11:58:54.600464+0000","last_unstale":"2026-02-20T11:58:56.565278+0000","last_undegraded":"2026-02-20T11:58:56.565278+0000","last_fullsized":"2026-02-20T11:58:56.565278+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566493+0000","last_change":"2026-02-20T11:58:56.568656+0000","last_active":"2026-02-20T11:58:56.566493+0000","last_peered":"2026-02-20T11:58:56.566493+0000","last_clean":"2026-02-20T11:58:56.566493+0000","last_became_active":"2026-02-20T11:58:54.590636+0000","last_became_peered":"2026-02-20T11:58:54.590636+0000","last_unstale":"2026-02-20T11:58:56.566493+0000","last_undegraded":"2026-02-20T11:58:56.566493+0000","last_fullsized":"2026-02-20T11:58:56.566493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'1","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566797+0000","last_change":"2026-02-20T11:58:56.566797+0000","last_active":"2026-02-20T11:58:56.566797+0000","last_peered":"2026-02-20T11:58:56.566797+0000","last_clean":"2026-02-20T11:58:56.566797+0000","last_became_active":"2026-02-20T11:58:54.585367+0000","last_became_peered":"2026-02-20T11:58:54.585367+0000","last_unstale":"2026-02-20T11:58:56.566797+0000","last_undegraded":"2026-02-20T11:58:56.566797+0000","last_fullsized":"2026-02-20T11:58:56.566797+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.563392+0000","last_change":"2026-02-20T11:58:56.566131+0000","last_active":"2026-02-20T11:58:56.563392+0000","last_peered":"2026-02-20T11:58:56.563392+0000","last_clean":"2026-02-20T11:58:56.563392+0000","last_became_active":"2026-02-20T11:58:54.586960+0000","last_became_peered":"2026-02-20T11:58:54.586960+0000","last_unstale":"2026-02-20T11:58:56.563392+0000","last_undegraded":"2026-02-20T11:58:56.563392+0000","last_fullsized":"2026-02-20T11:58:56.563392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.564188+0000","last_change":"2026-02-20T11:58:56.566676+0000","last_active":"2026-02-20T11:58:56.564188+0000","last_peered":"2026-02-20T11:58:56.564188+0000","last_clean":"2026-02-20T11:58:56.564188+0000","last_became_active":"2026-02-20T11:58:54.601082+0000","last_became_peered":"2026-02-20T11:58:54.601082+0000","last_unstale":"2026-02-20T11:58:56.564188+0000","last_undegraded":"2026-02-20T11:58:56.564188+0000","last_fullsized":"2026-02-20T11:58:56.564188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565521+0000","last_change":"2026-02-20T11:58:56.567835+0000","last_active":"2026-02-20T11:58:56.565521+0000","last_peered":"2026-02-20T11:58:56.565521+0000","last_clean":"2026-02-20T11:58:56.565521+0000","last_became_active":"2026-02-20T11:58:54.590960+0000","last_became_peered":"2026-02-20T11:58:54.590960+0000","last_unstale":"2026-02-20T11:58:56.565521+0000","last_undegraded":"2026-02-20T11:58:56.565521+0000","last_fullsized":"2026-02-20T11:58:56.565521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":91,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.564357+0000","last_change":"2026-02-20T11:58:48.113220+0000","last_active":"2026-02-20T11:58:56.564357+0000","last_peered":"2026-02-20T11:58:56.564357+0000","last_clean":"2026-02-20T11:58:56.564357+0000","last_became_active":"2026-02-20T11:58:48.102892+0000","last_became_peered":"2026-02-20T11:58:48.102892+0000","last_unstale":"2026-02-20T11:58:56.564357+0000","last_undegraded":"2026-02-20T11:58:56.564357+0000","last_fullsized":"2026-02-20T11:58:56.564357+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:47.064196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:47.064196+0000","last_clean_scrub_stamp":"2026-02-20T11:58:47.064196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672972,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165324,"statfs":{"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672972,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":440,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166216,"statfs":{"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1176,"kb_used_data":1176,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165480,"statfs":{"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1276,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165380,"statfs":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:59:00.560 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:59:00.636 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.629+0000 7f2c1facb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.639 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.634+0000 7f2c1facb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.749 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:59:00.749 INFO:teuthology.orchestra.run.trial153.stderr:dumped all 2026-02-20T11:59:00.758 INFO:teuthology.orchestra.run.trial153.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-20T11:59:00.721346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4224,"kb_used_data":4224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662400,"statfs":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.172321"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565536+0000","last_change":"2026-02-20T11:58:56.567811+0000","last_active":"2026-02-20T11:58:56.565536+0000","last_peered":"2026-02-20T11:58:56.565536+0000","last_clean":"2026-02-20T11:58:56.565536+0000","last_became_active":"2026-02-20T11:58:54.614068+0000","last_became_peered":"2026-02-20T11:58:54.614068+0000","last_unstale":"2026-02-20T11:58:56.565536+0000","last_undegraded":"2026-02-20T11:58:56.565536+0000","last_fullsized":"2026-02-20T11:58:56.565536+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566579+0000","last_change":"2026-02-20T11:58:56.568949+0000","last_active":"2026-02-20T11:58:56.566579+0000","last_peered":"2026-02-20T11:58:56.566579+0000","last_clean":"2026-02-20T11:58:56.566579+0000","last_became_active":"2026-02-20T11:58:54.590150+0000","last_became_peered":"2026-02-20T11:58:54.590150+0000","last_unstale":"2026-02-20T11:58:56.566579+0000","last_undegraded":"2026-02-20T11:58:56.566579+0000","last_fullsized":"2026-02-20T11:58:56.566579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565278+0000","last_change":"2026-02-20T11:58:56.567633+0000","last_active":"2026-02-20T11:58:56.565278+0000","last_peered":"2026-02-20T11:58:56.565278+0000","last_clean":"2026-02-20T11:58:56.565278+0000","last_became_active":"2026-02-20T11:58:54.600464+0000","last_became_peered":"2026-02-20T11:58:54.600464+0000","last_unstale":"2026-02-20T11:58:56.565278+0000","last_undegraded":"2026-02-20T11:58:56.565278+0000","last_fullsized":"2026-02-20T11:58:56.565278+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566493+0000","last_change":"2026-02-20T11:58:56.568656+0000","last_active":"2026-02-20T11:58:56.566493+0000","last_peered":"2026-02-20T11:58:56.566493+0000","last_clean":"2026-02-20T11:58:56.566493+0000","last_became_active":"2026-02-20T11:58:54.590636+0000","last_became_peered":"2026-02-20T11:58:54.590636+0000","last_unstale":"2026-02-20T11:58:56.566493+0000","last_undegraded":"2026-02-20T11:58:56.566493+0000","last_fullsized":"2026-02-20T11:58:56.566493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'1","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566797+0000","last_change":"2026-02-20T11:58:56.566797+0000","last_active":"2026-02-20T11:58:56.566797+0000","last_peered":"2026-02-20T11:58:56.566797+0000","last_clean":"2026-02-20T11:58:56.566797+0000","last_became_active":"2026-02-20T11:58:54.585367+0000","last_became_peered":"2026-02-20T11:58:54.585367+0000","last_unstale":"2026-02-20T11:58:56.566797+0000","last_undegraded":"2026-02-20T11:58:56.566797+0000","last_fullsized":"2026-02-20T11:58:56.566797+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.563392+0000","last_change":"2026-02-20T11:58:56.566131+0000","last_active":"2026-02-20T11:58:56.563392+0000","last_peered":"2026-02-20T11:58:56.563392+0000","last_clean":"2026-02-20T11:58:56.563392+0000","last_became_active":"2026-02-20T11:58:54.586960+0000","last_became_peered":"2026-02-20T11:58:54.586960+0000","last_unstale":"2026-02-20T11:58:56.563392+0000","last_undegraded":"2026-02-20T11:58:56.563392+0000","last_fullsized":"2026-02-20T11:58:56.563392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.564188+0000","last_change":"2026-02-20T11:58:56.566676+0000","last_active":"2026-02-20T11:58:56.564188+0000","last_peered":"2026-02-20T11:58:56.564188+0000","last_clean":"2026-02-20T11:58:56.564188+0000","last_became_active":"2026-02-20T11:58:54.601082+0000","last_became_peered":"2026-02-20T11:58:54.601082+0000","last_unstale":"2026-02-20T11:58:56.564188+0000","last_undegraded":"2026-02-20T11:58:56.564188+0000","last_fullsized":"2026-02-20T11:58:56.564188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565521+0000","last_change":"2026-02-20T11:58:56.567835+0000","last_active":"2026-02-20T11:58:56.565521+0000","last_peered":"2026-02-20T11:58:56.565521+0000","last_clean":"2026-02-20T11:58:56.565521+0000","last_became_active":"2026-02-20T11:58:54.590960+0000","last_became_peered":"2026-02-20T11:58:54.590960+0000","last_unstale":"2026-02-20T11:58:56.565521+0000","last_undegraded":"2026-02-20T11:58:56.565521+0000","last_fullsized":"2026-02-20T11:58:56.565521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":91,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.564357+0000","last_change":"2026-02-20T11:58:48.113220+0000","last_active":"2026-02-20T11:58:56.564357+0000","last_peered":"2026-02-20T11:58:56.564357+0000","last_clean":"2026-02-20T11:58:56.564357+0000","last_became_active":"2026-02-20T11:58:48.102892+0000","last_became_peered":"2026-02-20T11:58:48.102892+0000","last_unstale":"2026-02-20T11:58:56.564357+0000","last_undegraded":"2026-02-20T11:58:56.564357+0000","last_fullsized":"2026-02-20T11:58:56.564357+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:47.064196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:47.064196+0000","last_clean_scrub_stamp":"2026-02-20T11:58:47.064196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165324,"statfs":{"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":440,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166216,"statfs":{"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1176,"kb_used_data":1176,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165480,"statfs":{"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1276,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165380,"statfs":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:59:00.760 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-20T11:59:00.760 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-20T11:59:00.760 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-20T11:59:00.760 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-20T11:59:00.836 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.829+0000 7f9846764640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.838 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:00.833+0000 7f9846764640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:00.977 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:59:00.977 INFO:teuthology.orchestra.run.trial153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-20T11:59:00.986 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-20T11:59:00.986 INFO:teuthology.run_tasks:Running task thrashosds... 2026-02-20T11:59:01.003 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-20T11:59:01.004 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-perezjos-crimson-only-19-02-2026-PR64439'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '445d5c9ba18a4040e70e25303230c36c77773acb'}} 2026-02-20T11:59:01.004 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-20T11:59:01.004 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2026-02-20T11:59:01.081 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:01.073+0000 7f5980b5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:01.084 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:01.079+0000 7f5980b5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:01.208 INFO:tasks.thrashosds:Beginning thrashosds... 2026-02-20T11:59:01.208 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-20T11:59:01.208 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:59:01.285 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:01.277+0000 7fa10930b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:01.287 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:01.283+0000 7fa10930b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:01.398 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:59:01.398 INFO:teuthology.orchestra.run.trial153.stderr:dumped all 2026-02-20T11:59:01.406 INFO:teuthology.orchestra.run.trial153.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-20T11:59:00.721346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4224,"kb_used_data":4224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662400,"statfs":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.172321"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565536+0000","last_change":"2026-02-20T11:58:56.567811+0000","last_active":"2026-02-20T11:58:56.565536+0000","last_peered":"2026-02-20T11:58:56.565536+0000","last_clean":"2026-02-20T11:58:56.565536+0000","last_became_active":"2026-02-20T11:58:54.614068+0000","last_became_peered":"2026-02-20T11:58:54.614068+0000","last_unstale":"2026-02-20T11:58:56.565536+0000","last_undegraded":"2026-02-20T11:58:56.565536+0000","last_fullsized":"2026-02-20T11:58:56.565536+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566579+0000","last_change":"2026-02-20T11:58:56.568949+0000","last_active":"2026-02-20T11:58:56.566579+0000","last_peered":"2026-02-20T11:58:56.566579+0000","last_clean":"2026-02-20T11:58:56.566579+0000","last_became_active":"2026-02-20T11:58:54.590150+0000","last_became_peered":"2026-02-20T11:58:54.590150+0000","last_unstale":"2026-02-20T11:58:56.566579+0000","last_undegraded":"2026-02-20T11:58:56.566579+0000","last_fullsized":"2026-02-20T11:58:56.566579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565278+0000","last_change":"2026-02-20T11:58:56.567633+0000","last_active":"2026-02-20T11:58:56.565278+0000","last_peered":"2026-02-20T11:58:56.565278+0000","last_clean":"2026-02-20T11:58:56.565278+0000","last_became_active":"2026-02-20T11:58:54.600464+0000","last_became_peered":"2026-02-20T11:58:54.600464+0000","last_unstale":"2026-02-20T11:58:56.565278+0000","last_undegraded":"2026-02-20T11:58:56.565278+0000","last_fullsized":"2026-02-20T11:58:56.565278+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566493+0000","last_change":"2026-02-20T11:58:56.568656+0000","last_active":"2026-02-20T11:58:56.566493+0000","last_peered":"2026-02-20T11:58:56.566493+0000","last_clean":"2026-02-20T11:58:56.566493+0000","last_became_active":"2026-02-20T11:58:54.590636+0000","last_became_peered":"2026-02-20T11:58:54.590636+0000","last_unstale":"2026-02-20T11:58:56.566493+0000","last_undegraded":"2026-02-20T11:58:56.566493+0000","last_fullsized":"2026-02-20T11:58:56.566493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'1","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566797+0000","last_change":"2026-02-20T11:58:56.566797+0000","last_active":"2026-02-20T11:58:56.566797+0000","last_peered":"2026-02-20T11:58:56.566797+0000","last_clean":"2026-02-20T11:58:56.566797+0000","last_became_active":"2026-02-20T11:58:54.585367+0000","last_became_peered":"2026-02-20T11:58:54.585367+0000","last_unstale":"2026-02-20T11:58:56.566797+0000","last_undegraded":"2026-02-20T11:58:56.566797+0000","last_fullsized":"2026-02-20T11:58:56.566797+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.563392+0000","last_change":"2026-02-20T11:58:56.566131+0000","last_active":"2026-02-20T11:58:56.563392+0000","last_peered":"2026-02-20T11:58:56.563392+0000","last_clean":"2026-02-20T11:58:56.563392+0000","last_became_active":"2026-02-20T11:58:54.586960+0000","last_became_peered":"2026-02-20T11:58:54.586960+0000","last_unstale":"2026-02-20T11:58:56.563392+0000","last_undegraded":"2026-02-20T11:58:56.563392+0000","last_fullsized":"2026-02-20T11:58:56.563392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.564188+0000","last_change":"2026-02-20T11:58:56.566676+0000","last_active":"2026-02-20T11:58:56.564188+0000","last_peered":"2026-02-20T11:58:56.564188+0000","last_clean":"2026-02-20T11:58:56.564188+0000","last_became_active":"2026-02-20T11:58:54.601082+0000","last_became_peered":"2026-02-20T11:58:54.601082+0000","last_unstale":"2026-02-20T11:58:56.564188+0000","last_undegraded":"2026-02-20T11:58:56.564188+0000","last_fullsized":"2026-02-20T11:58:56.564188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565521+0000","last_change":"2026-02-20T11:58:56.567835+0000","last_active":"2026-02-20T11:58:56.565521+0000","last_peered":"2026-02-20T11:58:56.565521+0000","last_clean":"2026-02-20T11:58:56.565521+0000","last_became_active":"2026-02-20T11:58:54.590960+0000","last_became_peered":"2026-02-20T11:58:54.590960+0000","last_unstale":"2026-02-20T11:58:56.565521+0000","last_undegraded":"2026-02-20T11:58:56.565521+0000","last_fullsized":"2026-02-20T11:58:56.565521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":91,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.564357+0000","last_change":"2026-02-20T11:58:48.113220+0000","last_active":"2026-02-20T11:58:56.564357+0000","last_peered":"2026-02-20T11:58:56.564357+0000","last_clean":"2026-02-20T11:58:56.564357+0000","last_became_active":"2026-02-20T11:58:48.102892+0000","last_became_peered":"2026-02-20T11:58:48.102892+0000","last_unstale":"2026-02-20T11:58:56.564357+0000","last_undegraded":"2026-02-20T11:58:56.564357+0000","last_fullsized":"2026-02-20T11:58:56.564357+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:47.064196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:47.064196+0000","last_clean_scrub_stamp":"2026-02-20T11:58:47.064196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165324,"statfs":{"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":440,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166216,"statfs":{"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1176,"kb_used_data":1176,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165480,"statfs":{"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1276,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165380,"statfs":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:59:01.408 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T11:59:01.485 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:01.478+0000 7f07b910b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:01.487 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:01.483+0000 7f07b910b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:01.600 INFO:teuthology.orchestra.run.trial153.stdout: 2026-02-20T11:59:01.600 INFO:teuthology.orchestra.run.trial153.stderr:dumped all 2026-02-20T11:59:01.610 INFO:teuthology.orchestra.run.trial153.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2026-02-20T11:59:00.721346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":208666624,"kb_used":4224,"kb_used_data":4224,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":208662400,"statfs":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.172321"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565536+0000","last_change":"2026-02-20T11:58:56.567811+0000","last_active":"2026-02-20T11:58:56.565536+0000","last_peered":"2026-02-20T11:58:56.565536+0000","last_clean":"2026-02-20T11:58:56.565536+0000","last_became_active":"2026-02-20T11:58:54.614068+0000","last_became_peered":"2026-02-20T11:58:54.614068+0000","last_unstale":"2026-02-20T11:58:56.565536+0000","last_undegraded":"2026-02-20T11:58:56.565536+0000","last_fullsized":"2026-02-20T11:58:56.565536+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566579+0000","last_change":"2026-02-20T11:58:56.568949+0000","last_active":"2026-02-20T11:58:56.566579+0000","last_peered":"2026-02-20T11:58:56.566579+0000","last_clean":"2026-02-20T11:58:56.566579+0000","last_became_active":"2026-02-20T11:58:54.590150+0000","last_became_peered":"2026-02-20T11:58:54.590150+0000","last_unstale":"2026-02-20T11:58:56.566579+0000","last_undegraded":"2026-02-20T11:58:56.566579+0000","last_fullsized":"2026-02-20T11:58:56.566579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565278+0000","last_change":"2026-02-20T11:58:56.567633+0000","last_active":"2026-02-20T11:58:56.565278+0000","last_peered":"2026-02-20T11:58:56.565278+0000","last_clean":"2026-02-20T11:58:56.565278+0000","last_became_active":"2026-02-20T11:58:54.600464+0000","last_became_peered":"2026-02-20T11:58:54.600464+0000","last_unstale":"2026-02-20T11:58:56.565278+0000","last_undegraded":"2026-02-20T11:58:56.565278+0000","last_fullsized":"2026-02-20T11:58:56.565278+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566493+0000","last_change":"2026-02-20T11:58:56.568656+0000","last_active":"2026-02-20T11:58:56.566493+0000","last_peered":"2026-02-20T11:58:56.566493+0000","last_clean":"2026-02-20T11:58:56.566493+0000","last_became_active":"2026-02-20T11:58:54.590636+0000","last_became_peered":"2026-02-20T11:58:54.590636+0000","last_unstale":"2026-02-20T11:58:56.566493+0000","last_undegraded":"2026-02-20T11:58:56.566493+0000","last_fullsized":"2026-02-20T11:58:56.566493+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"17'1","reported_seq":18,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.566797+0000","last_change":"2026-02-20T11:58:56.566797+0000","last_active":"2026-02-20T11:58:56.566797+0000","last_peered":"2026-02-20T11:58:56.566797+0000","last_clean":"2026-02-20T11:58:56.566797+0000","last_became_active":"2026-02-20T11:58:54.585367+0000","last_became_peered":"2026-02-20T11:58:54.585367+0000","last_unstale":"2026-02-20T11:58:56.566797+0000","last_undegraded":"2026-02-20T11:58:56.566797+0000","last_fullsized":"2026-02-20T11:58:56.566797+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.563392+0000","last_change":"2026-02-20T11:58:56.566131+0000","last_active":"2026-02-20T11:58:56.563392+0000","last_peered":"2026-02-20T11:58:56.563392+0000","last_clean":"2026-02-20T11:58:56.563392+0000","last_became_active":"2026-02-20T11:58:54.586960+0000","last_became_peered":"2026-02-20T11:58:54.586960+0000","last_unstale":"2026-02-20T11:58:56.563392+0000","last_undegraded":"2026-02-20T11:58:56.563392+0000","last_fullsized":"2026-02-20T11:58:56.563392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.564188+0000","last_change":"2026-02-20T11:58:56.566676+0000","last_active":"2026-02-20T11:58:56.564188+0000","last_peered":"2026-02-20T11:58:56.564188+0000","last_clean":"2026-02-20T11:58:56.564188+0000","last_became_active":"2026-02-20T11:58:54.601082+0000","last_became_peered":"2026-02-20T11:58:54.601082+0000","last_unstale":"2026-02-20T11:58:56.564188+0000","last_undegraded":"2026-02-20T11:58:56.564188+0000","last_fullsized":"2026-02-20T11:58:56.564188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":17,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.565521+0000","last_change":"2026-02-20T11:58:56.567835+0000","last_active":"2026-02-20T11:58:56.565521+0000","last_peered":"2026-02-20T11:58:56.565521+0000","last_clean":"2026-02-20T11:58:56.565521+0000","last_became_active":"2026-02-20T11:58:54.590960+0000","last_became_peered":"2026-02-20T11:58:54.590960+0000","last_unstale":"2026-02-20T11:58:56.565521+0000","last_undegraded":"2026-02-20T11:58:56.565521+0000","last_fullsized":"2026-02-20T11:58:56.565521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:53.539398+0000","last_clean_scrub_stamp":"2026-02-20T11:58:53.539398+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":91,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-20T11:58:56.564357+0000","last_change":"2026-02-20T11:58:48.113220+0000","last_active":"2026-02-20T11:58:56.564357+0000","last_peered":"2026-02-20T11:58:56.564357+0000","last_clean":"2026-02-20T11:58:56.564357+0000","last_became_active":"2026-02-20T11:58:48.102892+0000","last_became_peered":"2026-02-20T11:58:48.102892+0000","last_unstale":"2026-02-20T11:58:56.564357+0000","last_undegraded":"2026-02-20T11:58:56.564357+0000","last_fullsized":"2026-02-20T11:58:56.564357+0000","mapping_epoch":12,"log_start":"13'48","ondisk_log_start":"13'48","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T11:58:47.064196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T11:58:47.064196+0000","last_clean_scrub_stamp":"2026-02-20T11:58:47.064196+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":48,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":213674622976,"available":213670297600,"internally_reserved":0,"allocated":4325376,"data_stored":4325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":3,"acting":3,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1332,"kb_used_data":1332,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165324,"statfs":{"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":440,"kb_used_data":440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52166216,"statfs":{"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1176,"kb_used_data":1176,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165480,"statfs":{"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52166656,"kb_used":1276,"kb_used_data":1276,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52165380,"statfs":{"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53418655744,"available":53417349120,"internally_reserved":0,"allocated":1306624,"data_stored":1306624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53418655744,"available":53418205184,"internally_reserved":0,"allocated":450560,"data_stored":450560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53418655744,"available":53417291776,"internally_reserved":0,"allocated":1363968,"data_stored":1363968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53418655744,"available":53417451520,"internally_reserved":0,"allocated":1204224,"data_stored":1204224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T11:59:01.611 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-20T11:59:01.611 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2026-02-20T11:59:01.687 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:01.680+0000 7f58a510b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:01.690 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:01.685+0000 7f58a510b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:01.802 INFO:teuthology.orchestra.run.trial153.stdout:epoch 18 2026-02-20T11:59:01.802 INFO:teuthology.orchestra.run.trial153.stdout:fsid 722e2839-9905-4d5d-898b-9fe4f73a126b 2026-02-20T11:59:01.802 INFO:teuthology.orchestra.run.trial153.stdout:created 2026-02-20T11:58:36.478281+0000 2026-02-20T11:59:01.802 INFO:teuthology.orchestra.run.trial153.stdout:modified 2026-02-20T11:58:56.551265+0000 2026-02-20T11:59:01.802 INFO:teuthology.orchestra.run.trial153.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2026-02-20T11:59:01.802 INFO:teuthology.orchestra.run.trial153.stdout:crush_version 5 2026-02-20T11:59:01.802 INFO:teuthology.orchestra.run.trial153.stdout:full_ratio 0.95 2026-02-20T11:59:01.802 INFO:teuthology.orchestra.run.trial153.stdout:backfillfull_ratio 0.9 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:nearfull_ratio 0.85 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:require_min_compat_client luminous 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:min_compat_client jewel 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:require_osd_release tentacle 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:stretch_mode_enabled false 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:allow_crimson=true 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 18 flags hashpspool,selfmanaged_snaps,crimson stripe_width 0 application rbd read_balance_score 1.50 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout: removed_snaps_queue [2~1] 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:max_osd 4 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:osd.0 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.153:6801/4091145385 v2:10.20.193.153:6800/4091145385 exists,up 27b61203-7e80-4ea1-9e43-38813bf3565e 2026-02-20T11:59:01.803 INFO:teuthology.orchestra.run.trial153.stdout:osd.1 up in weight 1 up_from 10 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.153:6810/3168964670 v2:10.20.193.153:6809/3168964670 exists,up ba0e97c8-b1b8-4469-b6fb-0dde4f3d6b24 2026-02-20T11:59:01.804 INFO:teuthology.orchestra.run.trial153.stdout:osd.2 up in weight 1 up_from 10 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.183:6805/2477092242 v2:10.20.193.183:6804/2477092242 exists,up c0b662d1-4522-4fc6-ad46-8c32656984cf 2026-02-20T11:59:01.804 INFO:teuthology.orchestra.run.trial153.stdout:osd.3 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.183:6806/527839342 v2:10.20.193.183:6803/527839342 exists,up acdba941-b115-46e2-ac92-9ddb4e52bd8e 2026-02-20T11:59:01.811 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.153:6801/4091145385 v2:10.20.193.153:6800/4091145385 exists,up 27b61203-7e80-4ea1-9e43-38813bf3565e', 'osd.1 up in weight 1 up_from 10 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.153:6810/3168964670 v2:10.20.193.153:6809/3168964670 exists,up ba0e97c8-b1b8-4469-b6fb-0dde4f3d6b24', 'osd.2 up in weight 1 up_from 10 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.183:6805/2477092242 v2:10.20.193.183:6804/2477092242 exists,up c0b662d1-4522-4fc6-ad46-8c32656984cf', 'osd.3 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) v2:10.20.193.183:6806/527839342 v2:10.20.193.183:6803/527839342 exists,up acdba941-b115-46e2-ac92-9ddb4e52bd8e'] 2026-02-20T11:59:01.812 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2026-02-20T11:59:01.886 INFO:teuthology.orchestra.run.trial153.stdout:{ 2026-02-20T11:59:01.886 INFO:teuthology.orchestra.run.trial153.stdout: "name": "mon.a", 2026-02-20T11:59:01.887 INFO:teuthology.orchestra.run.trial153.stdout: "cluster": "ceph", 2026-02-20T11:59:01.887 INFO:teuthology.orchestra.run.trial153.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2026-02-20T11:59:01.887 INFO:teuthology.orchestra.run.trial153.stdout: "admin_socket_mode": "", 2026-02-20T11:59:01.887 INFO:teuthology.orchestra.run.trial153.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2026-02-20T11:59:01.887 INFO:teuthology.orchestra.run.trial153.stdout: "auth_client_required": "cephx, none", 2026-02-20T11:59:01.887 INFO:teuthology.orchestra.run.trial153.stdout: "auth_cluster_required": "cephx", 2026-02-20T11:59:01.887 INFO:teuthology.orchestra.run.trial153.stdout: "auth_debug": "true", 2026-02-20T11:59:01.887 INFO:teuthology.orchestra.run.trial153.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2026-02-20T11:59:01.887 INFO:teuthology.orchestra.run.trial153.stdout: "auth_mon_ticket_ttl": "660.000000", 2026-02-20T11:59:01.887 INFO:teuthology.orchestra.run.trial153.stdout: "auth_service_required": "cephx", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "auth_service_ticket_ttl": "240.000000", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "auth_supported": "cephx", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_aio": "true", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_aio_max_queue_depth": "1024", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_aio_poll_ms": "250", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_aio_reap_max": "16", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_aio_submit_retry_initial_delay_us": "125", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_aio_submit_retry_max": "16", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_async_discard": "false", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_async_discard_max_pending": "1000000", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_async_discard_threads": "0", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_block_size": "4096", 2026-02-20T11:59:01.888 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_debug_aio": "false", 2026-02-20T11:59:01.889 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_debug_aio_log_age": "5.000000", 2026-02-20T11:59:01.889 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2026-02-20T11:59:01.889 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_debug_discard_sleep": "0", 2026-02-20T11:59:01.889 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_debug_inflight_ios": "false", 2026-02-20T11:59:01.889 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_discard_max_bytes": "10737418240", 2026-02-20T11:59:01.889 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_enable_discard": "false", 2026-02-20T11:59:01.889 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_flock_retry": "3", 2026-02-20T11:59:01.889 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_flock_retry_interval": "0.100000", 2026-02-20T11:59:01.889 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_inject_crash": "0", 2026-02-20T11:59:01.889 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_inject_crash_flush_delay": "2", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_ioring": "false", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_ioring_hipri": "false", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_ioring_sqthread_poll": "false", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_max_discard_length": "2147483648", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_nvme_unbind_from_kernel": "false", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_read_buffer_alignment": "4096", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_read_preallocated_huge_buffers": "", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_stalled_read_warn_threshold": "1", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bdev_type": "", 2026-02-20T11:59:01.890 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_alloc_size": "1048576", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_allocator": "hybrid", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_buffered_io": "true", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_check_for_zeros": "false", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_check_volume_selector_often": "false", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_check_volume_selector_on_umount": "false", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_compact_log_sync": "false", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_log_compact_min_size": "16777216", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_log_replay_check_allocations": "true", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_max_log_runway": "4194304", 2026-02-20T11:59:01.891 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_max_prefetch": "1048576", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_min_flush_size": "524288", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_min_log_runway": "1048576", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_replay_recovery": "false", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_replay_recovery_disable_compact": "false", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_shared_alloc_size": "65536", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_sync_write": "false", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluefs_wal_envelope_mode": "true", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_allocation_from_file": "true", 2026-02-20T11:59:01.892 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_allocator": "hybrid", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_allocator_lookup_policy": "auto", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_async_db_compaction": "true", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_bdev_label_multi": "true", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_bdev_label_require_all": "true", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_bitmapallocator_span_size": "1024", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_blobid_prealloc": "10240", 2026-02-20T11:59:01.893 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_block_create": "true", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_block_db_create": "false", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_block_db_path": "", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_block_db_size": "0", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_block_path": "", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_block_preallocate_file": "false", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_block_size": "107374182400", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_block_wal_create": "false", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_block_wal_path": "", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_block_wal_size": "100663296", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_bluefs": "true", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_bluefs_env_mirror": "false", 2026-02-20T11:59:01.894 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_bluefs_max_free": "10737418240", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_autotune": "true", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_autotune_interval": "5.000000", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_kv_ratio": "0.450000", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_meta_ratio": "0.450000", 2026-02-20T11:59:01.895 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_size": "0", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_size_hdd": "1073741824", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_size_ssd": "3221225472", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_trim_interval": "0.050000", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cache_type": "2q", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_clone_cow": "true", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_compression_algorithm": "snappy", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_compression_max_blob_size": "0", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_compression_min_blob_size": "0", 2026-02-20T11:59:01.896 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_compression_mode": "none", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_compression_required_ratio": "0.875000", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_csum_type": "crc32c", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_enforce_settings": "default", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_extent_map_encode_check": "false", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_freelist": "false", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_fsck_abort": "false", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_inject_read_err": "false", 2026-02-20T11:59:01.897 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_legacy_omap": "false", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_no_reuse_blocks": "false", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_omit_block_device_write": "false", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_omit_kv_commit": "false", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_onode_segmentation_random": "false", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_prefragment_max": "1048576", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_random_read_err": "0.000000", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_small_allocations": "0", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_default_buffered_read": "true", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_default_buffered_write": "false", 2026-02-20T11:59:01.898 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_deferred_batch_ops": "0", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_discard_on_mkfs": "true", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_elastic_shared_blobs": "true", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_extent_map_shard_max_size": "1200", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_extent_map_shard_min_size": "150", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_extent_map_shard_target_size": "500", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fail_eio": "false", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fragmentation_check_period": "3600", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_freelist_blocks_per_key": "128", 2026-02-20T11:59:01.899 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_on_mkfs": "true", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_on_mount": "false", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_on_mount_deep": "false", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_on_umount": "false", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_on_umount_deep": "false", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_quick_fix_threads": "2", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2026-02-20T11:59:01.900 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_gc_enable_blob_threshold": "0", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_gc_enable_total_threshold": "0", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_ignore_data_csum": "false", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_kvbackend": "rocksdb", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_log_collection_list_age": "60.000000", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_log_op_age": "5.000000", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_max_alloc_size": "0", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_max_blob_size": "0", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_max_blob_size_hdd": "65536", 2026-02-20T11:59:01.901 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_max_blob_size_ssd": "65536", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_max_defer_interval": "3.000000", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_max_deferred_txc": "32", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_min_alloc_size": "0", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_min_alloc_size_hdd": "4096", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_min_alloc_size_ssd": "4096", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_nid_prealloc": "1024", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_onode_segment_size": "0", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_prefer_deferred_size": "0", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_qfsck_on_mount": "true", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_recompression_min_gain": "1.200000", 2026-02-20T11:59:01.902 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_retry_disk_reads": "3", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_rocksdb_cf": "true", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_rocksdb_options_annex": "", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_slow_ops_warn_threshold": "1", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_spdk_coremask": "0x1", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_spdk_io_sleep": "5", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_spdk_max_io_completion": "0", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_spdk_mem": "512", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_sync_submit_transaction": "false", 2026-02-20T11:59:01.903 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_throttle_bytes": "67108864", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_throttle_cost_per_io": "0", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_throttle_trace_rate": "0.000000", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_tracing": "false", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_volume_selection_reserved": "0", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_warn_on_free_fragmentation": "0.800000", 2026-02-20T11:59:01.904 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_warn_on_legacy_statfs": "true", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_write_v2": "false", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_write_v2_random": "false", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "bluestore_zero_block_detection": "false", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "breakpad": "false", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "ceph_assert_supresssions": "", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "cephadm_path": "/usr/sbin/cephadm", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "cephfs_mirror_action_update_interval": "2", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "cephfs_mirror_directory_scan_interval": "10", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2026-02-20T11:59:01.905 INFO:teuthology.orchestra.run.trial153.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephfs_mirror_mount_timeout": "10", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephfs_mirror_perf_stats_prio": "5", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephsqlite_blocklist_dead_locker": "true", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephsqlite_lock_renewal_interval": "2000", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephx_cluster_require_signatures": "false", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephx_cluster_require_version": "2", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephx_require_signatures": "false", 2026-02-20T11:59:01.906 INFO:teuthology.orchestra.run.trial153.stdout: "cephx_require_version": "2", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "cephx_service_require_signatures": "false", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "cephx_service_require_version": "2", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "cephx_sign_messages": "true", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "chdir": "", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "client_acl_type": "", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "client_asio_thread_count": "2", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "client_cache_mid": "0.750000", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "client_cache_size": "16384", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "client_caps_release_delay": "5", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "client_check_pool_perm": "true", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "client_collect_and_send_global_metrics": "false", 2026-02-20T11:59:01.907 INFO:teuthology.orchestra.run.trial153.stdout: "client_debug_force_sync_read": "false", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_debug_getattr_caps": "false", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_debug_inject_features": "", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_debug_inject_tick_delay": "0", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_die_on_failed_dentry_invalidate": "true", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_die_on_failed_remount": "false", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_dirsize_rbytes": "true", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_force_lazyio": "false", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_fs": "", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_fscrypt_as": "true", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_fscrypt_dummy_encryption": "false", 2026-02-20T11:59:01.908 INFO:teuthology.orchestra.run.trial153.stdout: "client_inject_fixed_oldest_tid": "false", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_inject_release_failure": "false", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_inject_write_delay_secs": "0", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_max_inline_size": "4096", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_max_retries_on_remount_failure": "5", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_mds_namespace": "", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_metadata": "", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_mount_gid": "-1", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_mount_timeout": "300", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_mount_uid": "-1", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_mountpoint": "/", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_notify_timeout": "10", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_oc": "true", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_oc_max_dirty": "104857600", 2026-02-20T11:59:01.909 INFO:teuthology.orchestra.run.trial153.stdout: "client_oc_max_dirty_age": "5.000000", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_oc_max_objects": "1000", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_oc_size": "209715200", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_oc_target_dirty": "8388608", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_permissions": "true", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_quota": "true", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_quota_df": "true", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_readahead_max_bytes": "0", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_readahead_max_periods": "4", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_readahead_min": "131072", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_reconnect_stale": "false", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_respect_subvolume_snapshot_visibility": "false", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_shutdown_timeout": "30", 2026-02-20T11:59:01.910 INFO:teuthology.orchestra.run.trial153.stdout: "client_snapdir": ".snap", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "client_tick_interval": "1", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "client_trace": "", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "client_try_dentry_invalidate": "false", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "client_use_faked_inos": "false", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "client_use_random_mds": "false", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "clog_to_graylog": "false", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "clog_to_graylog_host": "127.0.0.1", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "clog_to_graylog_port": "12201", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "clog_to_monitors": "default=true", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "clog_to_syslog": "false", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "clog_to_syslog_level": "info", 2026-02-20T11:59:01.911 INFO:teuthology.orchestra.run.trial153.stdout: "cluster_addr": "-", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "cluster_network": "", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "cluster_network_interface": "", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "compressor_zlib_isal": "false", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "compressor_zlib_level": "5", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "compressor_zlib_winsize": "-15", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "compressor_zstd_level": "1", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "crash_dir": "/var/lib/ceph/crash", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_allow_pg_split": "true", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_bluestore_cpu_set": "", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_bluestore_num_threads": "6", 2026-02-20T11:59:01.912 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_cpu_num": "0", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_cpu_set": "", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_osd_obc_lru_size": "512", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_osd_scheduler_concurrency": "0", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_osd_stat_interval": "0", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_poll_mode": "false", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_reactor_backend": "", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_reactor_idle_poll_time_us": "200", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_reactor_io_latency_goal_ms": "0.000000", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crimson_reactor_task_quota_ms": "0.500000", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crush_location": "", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crush_location_hook": "", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "crush_location_hook_timeout": "10", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "d4n_writecache_enabled": "false", 2026-02-20T11:59:01.913 INFO:teuthology.orchestra.run.trial153.stdout: "daemonize": "false", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "daos_pool": "tank", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "dbstore_db_name_prefix": "dbstore", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "debug_alienstore": "0/5", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "debug_allow_any_pool_priority": "false", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "debug_asok": "1/5", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "debug_asok_assert_abort": "false", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "debug_asserts_on_shutdown": "true", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "debug_auth": "20/20", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "debug_bdev": "1/3", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "debug_bluefs": "1/5", 2026-02-20T11:59:01.914 INFO:teuthology.orchestra.run.trial153.stdout: "debug_bluestore": "1/5", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_bluestore_compression": "1/5", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_buffer": "0/1", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_ceph_dedup": "0/5", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_ceph_exporter": "1/5", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_cephfs_mirror": "0/5", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_cephsqlite": "0/5", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_client": "0/5", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_compressor": "1/5", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_context": "0/1", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_crimson_interrupt": "0/5", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_crush": "1/1", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_crypto": "1/5", 2026-02-20T11:59:01.915 INFO:teuthology.orchestra.run.trial153.stdout: "debug_cyanstore": "0/5", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_deliberately_leak_memory": "false", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_disable_randomized_ping": "false", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_dpdk": "1/5", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_eventtrace": "1/5", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_filer": "0/1", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_filestore": "1/3", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_finisher": "1/1", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_fuse": "1/5", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_heartbeat_testing_span": "0", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_heartbeatmap": "1/5", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_immutable_obj_cache": "0/5", 2026-02-20T11:59:01.916 INFO:teuthology.orchestra.run.trial153.stdout: "debug_javaclient": "1/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_journal": "1/3", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_journaler": "0/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_kstore": "1/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_lockdep": "0/1", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mclock": "1/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mds": "1/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mds_balancer": "1/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mds_locker": "1/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mds_log": "1/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mds_log_expire": "1/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mds_migrator": "1/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mds_quiesce": "3/5", 2026-02-20T11:59:01.917 INFO:teuthology.orchestra.run.trial153.stdout: "debug_memstore": "1/5", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mgr": "2/5", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mgrc": "1/5", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_mon": "20/20", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_monc": "0/10", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_ms": "1/1", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_none": "0/5", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_objclass": "0/5", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_objectcacher": "0/5", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_objecter": "0/1", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_optracker": "0/5", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_osd": "1/5", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_paxos": "20/20", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_perfcounter": "1/5", 2026-02-20T11:59:01.918 INFO:teuthology.orchestra.run.trial153.stdout: "debug_prioritycache": "1/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rados": "0/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rbd": "0/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rbd_mirror": "0/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rbd_pwl": "0/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rbd_replay": "0/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_refs": "0/0", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_reserver": "1/1", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw": "1/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw_access": "1/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw_bucket_logging": "1/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw_datacache": "1/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw_dbstore": "1/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw_dedup": "1/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw_flight": "1/5", 2026-02-20T11:59:01.919 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw_lifecycle": "1/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw_notification": "1/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw_restore": "1/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rgw_sync": "1/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_rocksdb": "4/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore": "0/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_backref": "0/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_cache": "0/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_cleaner": "0/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_device": "0/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_epm": "0/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_fixedkv_tree": "0/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_journal": "0/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_lba": "0/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_odata": "0/5", 2026-02-20T11:59:01.920 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_omap": "0/5", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_onode": "0/5", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_t": "0/5", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "debug_seastore_tm": "0/5", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "debug_striper": "0/1", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "debug_test": "0/5", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "debug_throttle": "1/1", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "debug_timer": "0/1", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "debug_tp": "0/5", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "debug_trace": "1/5", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "defer_client_eviction_on_laggy_osds": "false", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "device_failure_prediction_mode": "none", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "ec_extent_cache_size": "10485760", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "ec_pdw_write_mode": "0", 2026-02-20T11:59:01.921 INFO:teuthology.orchestra.run.trial153.stdout: "enable_availability_tracking": "false", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "err_to_graylog": "false", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "err_to_journald": "false", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "err_to_stderr": "true", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "err_to_syslog": "false", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "event_tracing": "false", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "exporter_addr": "0.0.0.0", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "exporter_cert_file": "", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "exporter_http_port": "9926", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "exporter_key_file": "", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "exporter_prio_limit": "5", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "exporter_sock_dir": "/var/run/ceph/", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "exporter_sort_metrics": "true", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "exporter_stats_period": "5", 2026-02-20T11:59:01.922 INFO:teuthology.orchestra.run.trial153.stdout: "fake_statfs_for_testing": "0", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "fatal_signal_handlers": "true", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filer_max_purge_ops": "10", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filer_max_truncate_ops": "128", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_apply_finisher_threads": "1", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_blackhole": "false", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_btrfs_clone_range": "true", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_btrfs_snap": "true", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_caller_concurrency": "10", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_collect_device_partition_information": "true", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_commit_timeout": "600.000000", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_debug_inject_read_err": "false", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_debug_omap_check": "false", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_debug_verify_split": "false", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_dump_file": "", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2026-02-20T11:59:01.923 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_expected_throughput_ops": "200.000000", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_fadvise": "true", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_fail_eio": "true", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_fd_cache_shards": "16", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_fd_cache_size": "128", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_fiemap": "false", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_fiemap_threshold": "4096", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_fsync_flushes_journal_data": "false", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_index_retry_probability": "0.000000", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_inject_stall": "0", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_journal_parallel": "false", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_journal_trailing": "false", 2026-02-20T11:59:01.924 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_journal_writeahead": "false", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_kill_at": "0", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_alloc_hint_size": "1048576", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_inline_xattr_size": "0", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_inline_xattr_size_other": "512", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_inline_xattrs": "0", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_inline_xattrs_other": "2", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_inline_xattrs_xfs": "10", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_sync_interval": "5.000000", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_xattr_value_size": "0", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_xattr_value_size_other": "1024", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_merge_threshold": "-10", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_min_sync_interval": "0.010000", 2026-02-20T11:59:01.925 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_odsync_write": "false", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_omap_backend": "rocksdb", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_omap_backend_path": "", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_omap_header_cache_size": "1024", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_ondisk_finisher_threads": "1", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_op_thread_suicide_timeout": "180", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_op_thread_timeout": "60", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_op_threads": "2", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_punch_hole": "false", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_queue_high_threshhold": "0.900000", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_queue_low_threshhold": "0.300000", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_queue_max_bytes": "104857600", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2026-02-20T11:59:01.926 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_queue_max_ops": "50", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_seek_data_hole": "false", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_sloppy_crc": "false", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_sloppy_crc_block_size": "65536", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_splice": "false", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_split_multiple": "2", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_split_rand_factor": "20", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_update_to": "1000", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_enable": "true", 2026-02-20T11:59:01.927 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_xfs_extsize": "false", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "filestore_zfs_snap": "false", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fio_dir": "/tmp/fio", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fsid": "722e2839-9905-4d5d-898b-9fe4f73a126b", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_allow_other": "true", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_atomic_o_trunc": "true", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_big_writes": "true", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_debug": "false", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_default_permissions": "false", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_disable_pagecache": "false", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_max_write": "0", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_multithreaded": "true", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_require_active_mds": "true", 2026-02-20T11:59:01.928 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_set_user_groups": "true", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_splice_move": "true", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_splice_read": "true", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_splice_write": "true", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_syncfs_on_mksnap": "true", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "fuse_use_invalidate_cb": "true", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "gss_target_name": "ceph", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "heartbeat_file": "", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "heartbeat_inject_failure": "0", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "heartbeat_interval": "5", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "host": "trial153", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_max_inflight_ops": "128", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_max_size": "1073741824", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_qos_bps_burst": "0", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2026-02-20T11:59:01.929 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_qos_bps_limit": "0", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_qos_iops_burst": "0", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_qos_iops_limit": "0", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "immutable_object_cache_watermark": "0.900000", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "inject_early_sigterm": "false", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "jaeger_agent_port": "6799", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "jaeger_tracing_enable": "false", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "journal_aio": "true", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "journal_align_min_size": "65536", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "journal_block_align": "true", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "journal_block_size": "4096", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "journal_dio": "true", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "journal_discard": "false", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "journal_force_aio": "false", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "journal_ignore_corruption": "false", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "journal_max_write_bytes": "10485760", 2026-02-20T11:59:01.930 INFO:teuthology.orchestra.run.trial153.stdout: "journal_max_write_entries": "100", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "journal_replay_from": "0", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "journal_throttle_high_multiple": "0.000000", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "journal_throttle_high_threshhold": "0.900000", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "journal_throttle_low_threshhold": "0.600000", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "journal_throttle_max_multiple": "0.000000", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "journal_write_header_frequency": "0", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "journal_zero_on_create": "false", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "journaler_prefetch_periods": "10", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "journaler_prezero_periods": "5", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "journaler_write_head_interval": "15", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "key": "", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "keyfile": "", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "librados_thread_count": "2", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "lockdep": "false", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "lockdep_force_backtrace": "false", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "log_coarse_timestamps": "true", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "log_flush_on_exit": "false", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "log_graylog_host": "127.0.0.1", 2026-02-20T11:59:01.931 INFO:teuthology.orchestra.run.trial153.stdout: "log_graylog_port": "12201", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "log_max_new": "1000", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "log_max_recent": "10000", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "log_stderr_prefix": "", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "log_stop_at_utilization": "0.970000", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "log_to_file": "true", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "log_to_graylog": "false", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "log_to_journald": "false", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "log_to_stderr": "false", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "log_to_syslog": "false", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "max_rotating_auth_attempts": "10", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "mds_action_on_write_error": "1", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "mds_allow_async_dirops": "true", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "mds_allow_batched_ops": "true", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "mds_alternate_name_max": "8192", 2026-02-20T11:59:01.932 INFO:teuthology.orchestra.run.trial153.stdout: "mds_asio_thread_count": "2", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_export_pin": "true", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_fragment_dirs": "true", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_fragment_interval": "5", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_fragment_size_max": "100000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_idle_threshold": "0.000000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_interval": "10", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_max": "-1", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_max_until": "-1", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_merge_size": "50", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_midchunk": "0.300000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_min_rebalance": "0.100000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_min_start": "0.200000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_minchunk": "0.001000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_mode": "0", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_need_max": "1.200000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_need_min": "0.800000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_overload_epochs": "2", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_replicate_threshold": "8000.000000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_sample_interval": "3.000000", 2026-02-20T11:59:01.933 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_split_bits": "3", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_split_rd": "25000.000000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_split_size": "10000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_split_wr": "10000.000000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_target_decay": "10.000000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_beacon_grace": "15.000000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_beacon_interval": "4.000000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_beacon_mon_down_grace": "60", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_memory_limit": "4294967296", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_mid": "0.700000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_quiesce_delay": "0", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_quiesce_sleep": "200", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_quiesce_splitauth": "true", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_quiesce_threshold": "524288", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_release_free_interval": "10", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_reservation": "0.050000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_trim_decay_rate": "1.000000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_trim_interval": "1", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cache_trim_threshold": "262144", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2026-02-20T11:59:01.934 INFO:teuthology.orchestra.run.trial153.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_client_delegate_inos_pct": "50", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_client_prealloc_inos": "1000", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_connect_bootstrapping": "false", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_damage_table_max_entries": "10000", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_debug_auth_pins": "false", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_debug_frag": "false", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_debug_scatterstat": "false", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_debug_subtrees": "false", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_decay_halflife": "5.000000", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_default_dir_hash": "2", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_defer_session_stale": "true", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_delay_journal_replay_for_testing": "0", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_deny_all_reconnect": "false", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_dir_keys_per_op": "16384", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_dir_max_commit_size": "10", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_dir_max_entries": "0", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_dir_prefetch": "true", 2026-02-20T11:59:01.935 INFO:teuthology.orchestra.run.trial153.stdout: "mds_dirstat_min_interval": "1.000000", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_dump_cache_after_rejoin": "false", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_dump_cache_on_map": "false", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_dump_cache_threshold_file": "0", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_early_reply": "true", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_enable_op_tracker": "true", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_enforce_unique_name": "true", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_export_ephemeral_distributed": "true", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_export_ephemeral_random": "true", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_export_ephemeral_random_max": "0.010000", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_extraordinary_events_dump_interval": "0", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_forward_all_requests_to_auth": "false", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_freeze_tree_timeout": "30.000000", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_fscrypt_last_block_max_size": "4096", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_go_bad_corrupt_dentry": "true", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_health_cache_threshold": "1.500000", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_health_summarize_threshold": "10", 2026-02-20T11:59:01.936 INFO:teuthology.orchestra.run.trial153.stdout: "mds_heartbeat_grace": "15.000000", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_heartbeat_reset_grace": "1000", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_inject_health_dummy": "false", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_inject_migrator_session_race": "false", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_inject_skip_replaying_inotable": "false", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_join_fs": "", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_journal_format": "1", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_after_journal_logs_flushed": "false", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_create_at": "0", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_dirfrag_at": "0", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_export_at": "0", 2026-02-20T11:59:01.937 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_import_at": "0", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_journal_at": "0", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_journal_expire_at": "0", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_journal_replay_at": "0", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_link_at": "0", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_mdstable_at": "0", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_openc_at": "0", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_rename_at": "0", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_kill_shutdown_at": "0", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_event_large_threshold": "524288", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_events_per_segment": "1024", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_max_events": "-1", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_max_segments": "128", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_minor_segments_per_major_segment": "16", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_pause": "false", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_segment_size": "0", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_skip_corrupt_events": "false", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_skip_unbounded_events": "false", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_trim_decay_rate": "1.000000", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_trim_threshold": "128", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_trim_upkeep_interval": "1000", 2026-02-20T11:59:01.938 INFO:teuthology.orchestra.run.trial153.stdout: "mds_log_warn_factor": "2.000000", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_max_caps_per_client": "1048576", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_max_completed_flushes": "100000", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_max_completed_requests": "100000", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_max_export_size": "20971520", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_max_file_recover": "32", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_max_purge_files": "64", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_max_purge_ops": "8192", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_max_scrub_ops_in_progress": "5", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_max_snaps_per_dir": "100", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_metrics_update_interval": "2", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_min_caps_per_client": "100", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_min_caps_working_set": "10000", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_mon_shutdown_timeout": "5.000000", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_numa_node": "-1", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_oft_prefetch_dirfrags": "false", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_op_complaint_time": "30.000000", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_op_history_duration": "600", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_op_history_size": "20", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_op_history_slow_op_size": "20", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_op_history_slow_op_threshold": "10", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_op_log_threshold": "5", 2026-02-20T11:59:01.939 INFO:teuthology.orchestra.run.trial153.stdout: "mds_ping_grace": "15", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_ping_interval": "5", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_recall_global_max_decay_threshold": "131072", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_recall_max_caps": "30000", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_recall_max_decay_rate": "1.500000", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_recall_max_decay_threshold": "131072", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_recall_warning_decay_rate": "60.000000", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_recall_warning_threshold": "262144", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_reconnect_timeout": "45.000000", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_replay_interval": "1.000000", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_replay_unsafe_with_closed_session": "false", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_request_load_average_decay_rate": "60.000000", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_root_ino_gid": "0", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_root_ino_uid": "0", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_scatter_nudge_interval": "5.000000", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_scrub_stats_review_period": "1", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_server_dispatch_client_request_delay": "0", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_session_blocklist_on_evict": "true", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_session_blocklist_on_timeout": "true", 2026-02-20T11:59:01.940 INFO:teuthology.orchestra.run.trial153.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_session_cache_liveness_magnitude": "10", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_session_cap_acquisition_throttle": "100000", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_session_metadata_threshold": "16777216", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_sessionmap_keys_per_op": "1024", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_shutdown_check": "0", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_skip_ino": "0", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_sleep_rank_change": "0.000000", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_snap_max_uid": "4294967294", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_snap_min_uid": "0", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_snap_rstat": "false", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_standby_replay_damaged": "false", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_symlink_recovery": "true", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_task_status_update_interval": "2.000000", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_thrash_exports": "0", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_thrash_fragments": "0", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_tick_interval": "5.000000", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_use_global_snaprealm_seq_for_subvol": "true", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_valgrind_exit": "false", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_verify_backtrace": "1", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_verify_scatter": "false", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_wipe_ino_prealloc": "false", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mds_wipe_sessions": "false", 2026-02-20T11:59:01.941 INFO:teuthology.orchestra.run.trial153.stdout: "mempool_debug": "false", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "memstore_debug_omit_block_device_write": "false", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "memstore_device_bytes": "1073741824", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "memstore_page_set": "false", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "memstore_page_size": "65536", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_client_bytes": "134217728", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_client_messages": "512", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_connect_retry_interval": "1.000000", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_disabled_modules": "", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_enable_op_tracker": "true", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_initial_modules": "iostat nfs", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_max_pg_creating": "1024", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_max_pg_num_change": "128", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_mds_bytes": "134217728", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_mds_messages": "128", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_mon_bytes": "134217728", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_mon_messages": "128", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_num_op_tracker_shard": "32", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_op_complaint_time": "30.000000", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_op_history_duration": "600", 2026-02-20T11:59:01.942 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_op_history_size": "20", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_op_history_slow_op_size": "20", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_op_log_threshold": "5", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_osd_bytes": "536870912", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_osd_messages": "8192", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_pool": "true", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_service_beacon_grace": "60.000000", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_standby_modules": "true", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_stats_period": "5", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_stats_threshold": "5", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_tick_period": "2", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mgr_ttl_cache_expire_seconds": "0", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_accept_timeout_factor": "2.000000", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_allow_pool_delete": "true", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_allow_pool_size_one": "true", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_auth_validate_all_caps": "true", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_bytes": "104857600", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_directed_command_retry": "2", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_hunt_interval": "3.000000", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_hunt_on_resend": "true", 2026-02-20T11:59:01.943 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_hunt_parallel": "3", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_log_interval": "1.000000", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_max_log_entries_per_message": "1000", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_ping_interval": "10.000000", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_ping_timeout": "30.000000", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_client_target_rank": "-1", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_clock_drift_allowed": "1.000000", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cluster_log_level": "debug", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cluster_log_to_file": "true", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cluster_log_to_graylog": "false", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cluster_log_to_graylog_port": "12201", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cluster_log_to_journald": "false", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cluster_log_to_stderr": "false", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cluster_log_to_syslog": "default=false", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_compact_on_bootstrap": "false", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_compact_on_start": "false", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_compact_on_trim": "true", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_con_tracker_persist_interval": "10", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_con_tracker_score_halflife": "43200", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_config_key_max_entry_size": "65536", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_cpu_threads": "4", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_crush_min_required_version": "hammer", 2026-02-20T11:59:01.944 INFO:teuthology.orchestra.run.trial153.stdout: "mon_daemon_bytes": "419430400", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_data_avail_crit": "5", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_data_avail_warn": "5", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_data_size_warn": "16106127360", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_block_osdmap_trim": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_deprecated_as_obsolete": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_dump_json": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_dump_transactions": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_extra_checks": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_no_initial_persistent_features": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_no_require_squid": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_no_require_tentacle": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_delta_reset_interval": "10.000000", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_dns_srv_name": "ceph-mon", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_down_added_grace": "180", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_down_mkfs_grace": "120", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_down_uptime_grace": "60", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_election_default_strategy": "1", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_election_timeout": "5.000000", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_elector_ping_divisor": "2", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_elector_ping_timeout": "2.000000", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_enable_op_tracker": "true", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_fake_pool_delete": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_force_quorum_join": "false", 2026-02-20T11:59:01.945 INFO:teuthology.orchestra.run.trial153.stdout: "mon_fsmap_prune_threshold": "300", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_globalid_prealloc": "10000", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_health_detail_to_clog": "false", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_health_log_update_period": "5", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_health_max_detail": "50", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_health_to_clog": "true", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_health_to_clog_interval": "600", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_host": "10.20.193.153", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_host_override": "", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_initial_members": "", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_inject_transaction_delay_max": "10.000000", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_keyvaluedb": "rocksdb", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_lease": "5.000000", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_lease_renew_interval_factor": "0.600000", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_log_full_interval": "50", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_log_max": "10000", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_log_max_summary": "50", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_max_log_entries_per_event": "4096", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_max_log_epochs": "500", 2026-02-20T11:59:01.946 INFO:teuthology.orchestra.run.trial153.stdout: "mon_max_mdsmap_epochs": "500", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_max_mgrmap_epochs": "500", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_max_nvmeof_epochs": "500", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_max_osd": "10000", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_max_pg_per_osd": "10000", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_max_pool_pg_num": "65536", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_max_snap_prune_per_epoch": "100", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_mds_blocklist_interval": "86400.000000", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_mds_force_trim_to": "0", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_mds_skip_sanity": "false", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_memory_autotune": "true", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_memory_target": "2147483648", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_mgr_beacon_grace": "30", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_mgr_digest_period": "5", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_mgr_inactive_grace": "60", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_mgr_mkfs_grace": "240", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_min_osdmap_epochs": "50", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_netsplit_grace_period": "9", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_nvmeofgw_beacon_grace": "7", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_nvmeofgw_beacons_till_ack": "15", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_nvmeofgw_delete_grace": "900", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_nvmeofgw_skip_failovers_interval": "16", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_nvmeofgw_wrong_map_ignore_sec": "15", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_op_complaint_time": "30", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_op_history_duration": "600", 2026-02-20T11:59:01.947 INFO:teuthology.orchestra.run.trial153.stdout: "mon_op_history_size": "20", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_op_history_slow_op_size": "20", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_op_history_slow_op_threshold": "10", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_op_log_threshold": "5", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_adjust_down_out_interval": "true", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_auto_mark_in": "false", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_auto_mark_new_in": "true", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_cache_size": "500", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_cache_size_min": "134217728", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_crush_smoke_test": "true", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_destroyed_out_interval": "600", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_down_out_interval": "600", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_down_out_subtree_limit": "rack", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_force_trim_to": "0", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_full_ratio": "0.950000", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_laggy_halflife": "3600", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_laggy_max_interval": "300", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_laggy_weight": "0.300000", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_max_initial_pgs": "1024", 2026-02-20T11:59:01.948 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_min_down_reporters": "2", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_min_in_ratio": "0.750000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_min_up_ratio": "0.300000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_nearfull_ratio": "0.850000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_prime_pg_temp": "true", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_report_timeout": "900", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_reporter_subtree_level": "osd", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_warn_num_repaired": "10", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osd_warn_op_age": "32.000000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osdmap_full_prune_enabled": "true", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osdmap_full_prune_interval": "2", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osdmap_full_prune_min": "15", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_osdmap_full_prune_txsize": "2", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_pg_stuck_threshold": "60", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_pg_warn_min_objects": "10000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_pg_warn_min_per_osd": "0", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_pg_warn_min_pool_objects": "1000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_pool_quota_crit_threshold": "0", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_pool_quota_warn_threshold": "0", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_probe_timeout": "2.000000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_reweight_max_change": "0.050000", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_reweight_max_osds": "4", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_reweight_min_bytes_per_osd": "10", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_reweight_min_pgs_per_osd": "4", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2026-02-20T11:59:01.949 INFO:teuthology.orchestra.run.trial153.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_scrub_interval": "86400", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_scrub_max_keys": "100", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_scrub_timeout": "300", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_session_timeout": "300", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_smart_report_timeout": "5", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_stat_smooth_intervals": "6", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_stretch_pool_min_size": "2", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_stretch_pool_size": "4", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_subscribe_interval": "86400.000000", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_sync_debug": "false", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_sync_max_payload_keys": "2000", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_sync_max_payload_size": "1048576", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_sync_provider_kill_at": "0", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_sync_requester_kill_at": "0", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_sync_timeout": "60.000000", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_target_pg_per_osd": "100", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_tick_interval": "5", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_timecheck_interval": "300.000000", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_timecheck_skew_interval": "30.000000", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_use_min_delay_socket": "false", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_older_version_delay": "604800", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_filestore_osds": "false", 2026-02-20T11:59:01.950 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_misplaced": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_older_version": "true", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_pool_no_app": "true", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_pool_no_app_grace": "300", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_pool_no_redundancy": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_on_too_few_osds": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "monmap": "", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "motr_tracing_enabled": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_op_threads": "3", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_buffer_size": "131072", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_cm": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_device_name": "", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_dscp": "96", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_enable_hugepage": "false", 2026-02-20T11:59:01.951 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_gid_idx": "0", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_local_gid": "", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_polling_us": "1000", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_port_num": "1", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_receive_buffers": "32768", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_receive_queue_len": "4096", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_roce_ver": "1", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_send_buffers": "1024", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_sl": "3", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_support_srq": "true", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_rdma_type": "ib", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_async_reap_threshold": "5", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_bind_before_connect": "false", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_bind_ipv4": "true", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_bind_ipv6": "false", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_bind_msgr1": "true", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_bind_msgr2": "true", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_bind_port_max": "7568", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_bind_port_min": "6800", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_bind_prefer_ipv4": "false", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_bind_retry_count": "3", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_bind_retry_delay": "5", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_blackhole_client": "false", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_blackhole_mds": "false", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_blackhole_mgr": "false", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_blackhole_mon": "false", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_blackhole_osd": "false", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_client_mode": "crc", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_client_throttle_retry_time_interval": "5000", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_cluster_mode": "crc", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_cluster_type": "", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_compress_secure": "false", 2026-02-20T11:59:01.952 INFO:teuthology.orchestra.run.trial153.stdout: "ms_connection_idle_timeout": "900", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_connection_ready_timeout": "10", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_crc_data": "true", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_crc_header": "true", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_die_on_bad_msg": "false", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_die_on_bug": "true", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_die_on_old_message": "true", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_die_on_skipped_message": "false", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_die_on_unhandled_msg": "false", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dispatch_throttle_bytes": "104857600", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_coremask": "0xF", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_debug_allow_loopback": "false", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_devs_allowlist": "", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_enable_tso": "true", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_host_ipv4_addr": "", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_hugepages": "", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_hw_flow_control": "true", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_lro": "true", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_memory_channel": "4", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_pmd": "", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_port_id": "0", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dump_corrupt_message_level": "1", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_dump_on_send": "false", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_initial_backoff": "0.200000", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_inject_delay_max": "1.000000", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_inject_delay_probability": "0.000000", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_inject_delay_type": "", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_inject_internal_delays": "0.000000", 2026-02-20T11:59:01.953 INFO:teuthology.orchestra.run.trial153.stdout: "ms_inject_network_congestion": "0", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_inject_socket_failures": "0", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_learn_addr_from_peer": "true", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_max_accept_failures": "4", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_max_backoff": "15.000000", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_mon_client_mode": "crc", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_mon_cluster_mode": "crc", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_mon_service_mode": "crc", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_osd_compress_min_size": "1024", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_osd_compress_mode": "none", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_osd_compression_algorithm": "snappy", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_pq_min_cost": "65536", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_public_type": "", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_service_mode": "crc", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_tcp_listen_backlog": "512", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_tcp_nodelay": "true", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_tcp_prefetch_max_size": "65536", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_tcp_rcvbuf": "0", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_time_events_min_wait_interval": "1000", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "ms_type": "async+posix", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "no_config_file": "false", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "nvmeof_mon_client_connect_panic": "30", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "nvmeof_mon_client_tick_period": "1", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "objecter_completion_locks_per_session": "32", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "objecter_debug_inject_relock_delay": "false", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "objecter_inflight_op_bytes": "104857600", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "objecter_inflight_ops": "1024", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "objecter_inject_no_watch_ping": "false", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "objecter_retry_writes_after_first_reply": "false", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "objecter_tick_interval": "5.000000", 2026-02-20T11:59:01.954 INFO:teuthology.orchestra.run.trial153.stdout: "objecter_timeout": "10.000000", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "objectstore_blackhole": "false", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "openssl_conf": "", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_agent_delay_time": "5.000000", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_agent_hist_halflife": "1000", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_agent_max_low_ops": "2", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_agent_max_ops": "4", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_agent_min_evict_effort": "0.100000", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_agent_quantize_effort": "0.100000", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_agent_slop": "0.020000", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_aggregated_slow_ops_logging": "true", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_allow_recovery_below_min_size": "true", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_asio_thread_count": "2", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_async_recovery_min_cost": "100", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_auto_mark_unfound_lost": "false", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_backfill_retry_interval": "30.000000", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_backfill_scan_max": "512", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_backfill_scan_min": "64", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_backoff_on_degraded": "false", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_backoff_on_peering": "false", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_backoff_on_unfound": "true", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_beacon_report_interval": "300", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_bench_duration": "30", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_bench_large_size_max_throughput": "104857600", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_bench_max_block_size": "67108864", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_bench_small_size_max_iops": "100", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_blkin_trace_all": "false", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_blocked_scrub_grace_period": "120", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_check_for_log_corruption": "false", 2026-02-20T11:59:01.955 INFO:teuthology.orchestra.run.trial153.stdout: "osd_check_max_object_name_len_on_startup": "true", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_class_update_on_start": "true", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_client_message_cap": "256", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_client_message_size_cap": "524288000", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_client_op_priority": "63", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_client_watch_timeout": "30", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_command_max_records": "256", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_compact_on_start": "false", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_copyfrom_max_chunk": "8388608", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_crush_chooseleaf_type": "0", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_crush_initial_weight": "-1.000000", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_crush_update_on_start": "true", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_crush_update_weight_set": "true", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_drop_ping_duration": "0", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_drop_ping_probability": "0.000000", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_feed_pullee": "-1", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_inject_copyfrom_error": "false", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_misdirected_ops": "false", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_no_acting_change": "false", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_no_purge_strays": "false", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_op_order": "false", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_pg_log_writeout": "false", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_pretend_recovery_active": "false", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_random_push_read_error": "0.000000", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_shutdown": "false", 2026-02-20T11:59:01.956 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_trim_objects": "false", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_verify_cached_snaps": "false", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_verify_missing_on_start": "false", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_verify_snaps": "false", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_debug_verify_stray_on_activate": "false", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_deep_scrub_interval": "604800.000000", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_deep_scrub_interval_cv": "0.200000", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_deep_scrub_keys": "1024", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_deep_scrub_stride": "4194304", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_default_data_pool_replay_window": "5", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_default_notify_timeout": "30", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_delete_sleep": "0.000000", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_delete_sleep_hdd": "5.000000", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_delete_sleep_hybrid": "1.000000", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_delete_sleep_ssd": "1.000000", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_discard_disconnected_ops": "true", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_ec_partial_reads": "true", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_enable_op_tracker": "true", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_extblkdev_plugins": "vdo", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_failsafe_full_ratio": "0.970000", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_fast_fail_on_connection_refused": "true", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_fast_info": "true", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_fast_shutdown": "true", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_fast_shutdown_notify_mon": "true", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_fast_shutdown_timeout": "15", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_find_best_info_ignore_history_les": "false", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_force_auth_primary_missing_objects": "100", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2026-02-20T11:59:01.957 INFO:teuthology.orchestra.run.trial153.stdout: "osd_function_tracing": "false", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_heartbeat_grace": "20", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_heartbeat_interval": "6", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_heartbeat_min_peers": "10", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_heartbeat_min_size": "2000", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_heartbeat_stale": "600", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_hit_set_max_size": "100000", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_hit_set_min_size": "1000", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_hit_set_namespace": ".ceph-internal", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_ignore_stale_divergent_priors": "false", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_inject_failure_on_pg_removal": "false", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_journal_flush_on_shutdown": "true", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_journal_size": "5120", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_kill_backfill_at": "0", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_loop_before_reset_tphandle": "64", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_map_cache_size": "50", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_map_dedup": "true", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_map_message_max": "40", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_map_message_max_bytes": "10485760", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_map_share_max_epochs": "40", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_attr_name_len": "100", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_attr_size": "0", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_backfills": "1", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_markdown_count": "5", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_markdown_period": "600", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_object_name_len": "2048", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_object_namespace_len": "256", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_object_size": "134217728", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_omap_entries_per_request": "1024", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_pg_blocked_by": "16", 2026-02-20T11:59:01.958 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_pg_log_entries": "2", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_pgls": "1024", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_push_cost": "8388608", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_push_objects": "10", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_scrubs": "3", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_trimming_pgs": "2", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_write_op_reply_len": "64", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_max_write_size": "90", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_iops_capacity_low_threshold_hdd": "50.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_iops_capacity_low_threshold_ssd": "1000.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_override_recovery_settings": "false", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_profile": "balanced", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_scheduler_client_wgt": "1", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mclock_skip_benchmark": "false", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_memory_base": "805306368", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_memory_cache_min": "134217728", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_memory_cache_resize_interval": "1.000000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_memory_expected_fragmentation": "0.150000", 2026-02-20T11:59:01.959 INFO:teuthology.orchestra.run.trial153.stdout: "osd_memory_target": "4294967296", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_memory_target_autotune": "false", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_min_pg_log_entries": "1", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_min_recovery_priority": "0", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mon_heartbeat_interval": "30", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mon_report_interval": "5", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mon_report_max_in_flight": "2", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_mon_shutdown_timeout": "5.000000", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_num_cache_shards": "32", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_num_op_tracker_shard": "32", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_numa_auto_affinity": "true", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_numa_node": "-1", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_numa_prefer_iface": "true", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_object_clean_region_max_num_intervals": "10", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_objecter_finishers": "1", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_objectstore": "bluestore", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_objectstore_fuse": "false", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_objectstore_ideal_list_max": "64", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_objectstore_tracing": "false", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_complaint_time": "30.000000", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_history_duration": "600", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_history_size": "20", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_history_slow_op_size": "20", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_log_threshold": "5", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_num_shards": "0", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_num_shards_hdd": "1", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_num_shards_ssd": "8", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_num_threads_per_shard": "0", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_num_threads_per_shard_hdd": "5", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_pq_min_cost": "65536", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_queue": "mclock_scheduler", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_queue_cut_off": "high", 2026-02-20T11:59:01.960 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_thread_suicide_timeout": "150", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_op_thread_timeout": "15", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_open_classes_on_start": "true", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_os_flags": "0", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_peering_op_priority": "255", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_delete_cost": "1048576", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_delete_priority": "5", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_log_dups_tracked": "3000", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_log_trim_max": "10000", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_log_trim_min": "0", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_object_context_cache_count": "64", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_cache_min_evict_age": "0", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_cache_min_flush_age": "0", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_crimson": "true", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_crush_rule": "-1", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_ec_fast_read": "false", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_erasure_code_profile": "plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_flag_bulk": "false", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_flag_ec_optimizations": "false", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_flag_hashpspool": "true", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_flag_nodelete": "false", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_flag_nopgchange": "false", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_flag_nosizechange": "false", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_flags": "0", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_min_size": "2", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2026-02-20T11:59:01.961 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_pg_num": "32", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_pgp_num": "0", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_read_ratio": "70", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_size": "3", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_default_type": "replicated", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_erasure_code_stripe_unit": "0", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_pool_use_gmt_hitset": "true", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_push_per_object_cost": "1000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_read_ec_check_for_errors": "false", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recover_clone_overlap": "true", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recover_clone_overlap_limit": "10", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_cost": "20971520", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_delay_start": "0.000000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_max_active": "0", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_max_active_hdd": "3", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_max_active_ssd": "10", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_max_chunk": "8388608", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_max_single_start": "1", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_op_priority": "3", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_op_warn_multiple": "16", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_priority": "5", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_retry_interval": "30.000000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_sleep": "0.000000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_sleep_degraded": "0.000000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_sleep_degraded_hdd": "0.100000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_sleep_degraded_hybrid": "0.025000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_sleep_degraded_ssd": "0.000000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_sleep_hdd": "0.100000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_recovery_sleep_ssd": "0.000000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_requested_scrub_priority": "5", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_rollback_to_cluster_snap": "", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_auto_repair": "false", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_auto_repair_num_errors": "5", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_backoff_ratio": "0.660000", 2026-02-20T11:59:01.962 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_begin_hour": "0", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_begin_week_day": "0", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_chunk_max": "15", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_chunk_min": "5", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_cost": "52428800", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_disable_reservation_queuing": "false", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_during_recovery": "false", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_end_hour": "0", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_end_week_day": "0", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_event_cost": "4096", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_extended_sleep": "0.000000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_invalid_stats": "true", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_load_threshold": "10.000000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_max_interval": "604800.000000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_max_preemptions": "5", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_min_interval": "86400.000000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_priority": "5", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_retry_after_noscrub": "60", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_retry_delay": "30", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_retry_new_interval": "10", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_retry_pg_state": "60", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_retry_trimming": "10", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_scrub_sleep": "0.000000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_shallow_scrub_chunk_max": "100", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_shallow_scrub_chunk_min": "50", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_shutdown_pgref_assert": "false", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_skip_check_past_interval_bounds": "false", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_skip_data_digest": "false", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_smart_report_timeout": "5", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_snap_trim_cost": "1048576", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_snap_trim_priority": "5", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_snap_trim_sleep": "0.000000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_stats_update_period_not_scrubbing": "120", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_stats_update_period_scrubbing": "15", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_target_transaction_size": "30", 2026-02-20T11:59:01.963 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tier_default_cache_hit_set_count": "4", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tier_default_cache_mode": "writeback", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tier_promote_max_objects_sec": "25", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_tracing": "false", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_use_stale_snap": "false", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "osdc_blkin_trace_all": "false", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_kill_at": "0", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_max_join_drift": "10", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_min": "500", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_min_wait": "0.050000", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_propose_interval": "1.000000", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_service_trim_max": "500", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_service_trim_max_multiplier": "20", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_service_trim_min": "10", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_stash_full_interval": "25", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_trim_max": "500", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "paxos_trim_min": "250", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "perf": "true", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "plugin_crypto_accelerator": "crypto_isal", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "plugin_dir": "/usr/lib64/ceph", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "pool_availability_update_interval": "1.000000", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "public_addr": "-", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "public_addrv": "", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "public_bind_addr": "-", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "public_network": "", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "public_network_interface": "", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "qat_compressor_busy_polling": "false", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "qat_compressor_enabled": "false", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "qat_compressor_session_max_number": "256", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "rados_mon_op_timeout": "0", 2026-02-20T11:59:01.964 INFO:teuthology.orchestra.run.trial153.stdout: "rados_osd_op_timeout": "0", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rados_replica_read_policy": "default", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rados_replica_read_policy_on_objclass": "false", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rados_tracing": "false", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_atime_update_interval": "60", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_balance_parent_reads": "false", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_balance_snap_reads": "false", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_blkin_trace_all": "false", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_blocklist_expire_seconds": "0", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_blocklist_on_break_lock": "true", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_cache": "true", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_cache_block_writes_upfront": "false", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_cache_max_dirty": "25165824", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_cache_max_dirty_age": "1.000000", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_cache_max_dirty_object": "0", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_cache_policy": "writearound", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_cache_size": "33554432", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_cache_target_dirty": "16777216", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_cache_writethrough_until_flush": "true", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_clone_copy_on_read": "false", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_compression_hint": "none", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_concurrent_management_ops": "10", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_config_pool_override_update_timestamp": "0", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_default_clone_format": "auto", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_default_data_pool": "", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_default_features": "61", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_default_format": "2", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_default_map_options": "", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_default_order": "22", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_default_pool": "rbd", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_default_stripe_count": "0", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_default_stripe_unit": "0", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_disable_zero_copy_writes": "true", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_discard_granularity_bytes": "65536", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_discard_on_zeroed_write_same": "true", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_enable_alloc_hint": "true", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_io_scheduler": "simple", 2026-02-20T11:59:01.965 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_commit_age": "5.000000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_max_payload_bytes": "16384", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_object_flush_age": "0.000000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_object_flush_bytes": "1048576", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_object_flush_interval": "0", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_order": "24", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_pool": "", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_journal_splay_width": "4", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_localize_parent_reads": "false", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_localize_snap_reads": "false", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_image_policy_type": "simple", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_image_state_check_interval": "30", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_memory_autotune": "true", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_memory_base": "805306368", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_memory_cache_min": "134217728", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_memory_target": "4294967296", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_perf_stats_prio": "5", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirroring_delete_delay": "0", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirroring_replay_delay": "0", 2026-02-20T11:59:01.966 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_move_to_trash_on_remove": "false", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_mtime_update_interval": "60", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_non_blocking_aio": "true", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_op_thread_timeout": "60", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_op_threads": "1", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_parent_cache_enabled": "false", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_persistent_cache_mode": "disabled", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_persistent_cache_path": "/tmp", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_persistent_cache_size": "1073741824", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_plugins": "", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_bps_burst": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_bps_burst_seconds": "1", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_bps_limit": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_exclude_ops": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_iops_burst": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_iops_burst_seconds": "1", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_iops_limit": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_read_bps_burst": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_read_bps_limit": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_read_iops_burst": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_read_iops_limit": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_schedule_tick_min": "50", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_write_bps_burst": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_write_bps_limit": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_write_iops_burst": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_qos_write_iops_limit": "0", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_quiesce_notification_attempts": "10", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_read_from_replica_policy": "default", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_readahead_max_bytes": "524288", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_readahead_trigger_requests": "10", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_request_timed_out_seconds": "30", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_skip_partial_discard": "true", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2026-02-20T11:59:01.967 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_tracing": "false", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_validate_names": "true", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rbd_validate_pool": "true", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "restapi_base_url": "", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "restapi_log_level": "", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_account_default_quota_max_objects": "-1", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_account_default_quota_max_size": "-1", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_acl_grants_max_num": "100", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_admin_entry": "admin", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_asio_assert_yielding": "false", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_backend_store": "rados", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_barbican_url": "", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_beast_enable_async": "true", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_counters_cache": "false", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_counters_cache_size": "10000", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_default_quota_max_size": "-1", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_eexist_override": "false", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_index_max_aio": "128", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_logging_obj_roll_time": "300", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_persistent_notif_num_shards": "11", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_quota_cache_size": "10000", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_quota_ttl": "600", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_bucket_sync_spawn_window": "20", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_cache_enabled": "true", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_cache_expiry_interval": "900", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_cache_lru_size": "25000", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_config_store": "rados", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_content_length_compat": "false", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_copy_obj_progress": "true", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_cors_rules_max_num": "100", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_cross_domain_policy": "", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_default_encryption_key": "", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_kmip_addr": "", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_kmip_ca_path": "", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_kmip_client_cert": "", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_kmip_client_key": "", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_kmip_kms_key_template": "", 2026-02-20T11:59:01.968 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_kmip_password": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_kmip_username": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_require_ssl": "true", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_backend": "vault", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_suppress_logs": "true", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_vault_addr": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_vault_auth": "token", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_vault_namespace": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_vault_prefix": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_vault_secret_engine": "transit", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_vault_ssl_cacert": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_vault_token_file": "", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_crypt_vault_verify_ssl": "true", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_curl_buffersize": "524288", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_curl_low_speed_limit": "1024", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_curl_low_speed_time": "300", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_curl_tcp_keepalive": "0", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_curl_wait_timeout_ms": "1000", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d3n_l1_fadvise": "4", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d3n_libaio_aio_num": "64", 2026-02-20T11:59:01.969 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d3n_libaio_aio_threads": "20", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_backend_address": "127.0.0.1:6379", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_cache_cleaning_interval": "1000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_l1_datacache_disk_reserve": "1073741824", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_l1_fadvise": "4", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_l1_write_open_flags": "4096", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_libaio_aio_num": "64", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_libaio_aio_threads": "20", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_d4n_local_rgw_address": "127.0.0.1:8000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_data_log_changes_size": "1000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_data_log_num_shards": "128", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_data_log_window": "30", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_data_notify_interval_msec": "0", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_data_sync_poll_interval": "20", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_data_sync_spawn_window": "20", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_debug_inject_set_olh_err": "0", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_default_data_log_backing": "fifo", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_default_realm_info_oid": "default.realm", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_default_region_info_oid": "default.region", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_default_zone_info_oid": "default.zone", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_defer_to_bucket_acls": "", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_delete_multi_obj_max_num": "1000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_disable_s3select": "false", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_admin_lim": "0.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_admin_res": "100.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_auth_lim": "0.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_auth_res": "200.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_data_lim": "0.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_data_res": "500.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_data_wgt": "500.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_metadata_res": "500.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2026-02-20T11:59:01.970 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dns_name": "", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dns_s3website_name": "", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dynamic_resharding": "true", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enable_gc_threads": "true", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enable_jwks_url_verification": "false", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enable_lc_threads": "true", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enable_mdsearch": "true", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enable_ops_log": "false", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enable_quota_threads": "true", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enable_restore_threads": "true", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enable_static_website": "false", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enable_usage_log": "false", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_enforce_swift_acls": "true", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_exit_timeout_secs": "120", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_expose_bucket": "false", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_extended_http_attrs": "", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_filter": "none", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_frontends": "beast port=7480", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_gc_max_concurrent_io": "10", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_gc_max_deferred": "50", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_gc_max_objs": "32", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_gc_max_queue_size": "134213632", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_gc_max_trim_chunk": "16", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_gc_obj_min_wait": "7200", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_gc_processor_max_time": "3600", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_gc_processor_period": "3600", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_get_obj_max_req_size": "4194304", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_get_obj_window_size": "16777216", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_graceful_stop": "false", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_healthcheck_disabling_path": "", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_http_notif_connection_timeout": "5", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_http_notif_max_inflight": "8192", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_http_notif_message_timeout": "10", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ignore_get_invalid_range": "false", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_init_timeout": "300", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_kafka_connection_idle": "300", 2026-02-20T11:59:01.971 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_kafka_message_timeout": "5000", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_kafka_sleep_timeout": "10", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_accepted_admin_roles": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_accepted_reader_roles": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_admin_domain": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_admin_password": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_admin_password_path": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_admin_project": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_admin_tenant": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_admin_user": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_barbican_domain": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_barbican_password": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_barbican_project": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_barbican_tenant": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_barbican_user": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_implicit_tenants": "false", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_service_token_enabled": "false", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_token_cache_size": "10000", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_token_cache_ttl": "300", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_url": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_keystone_verify_ssl": "true", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lc_debug_interval": "-1", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lc_list_cnt": "1000", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lc_lock_max_time": "90", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lc_max_objs": "32", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lc_max_rules": "1000", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lc_max_worker": "3", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lc_max_wp_worker": "128", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lc_ordered_list_threshold": "500", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lc_thread_delay": "0", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ldap_dnattr": "uid", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ldap_searchfilter": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ldap_uri": "", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lfuda_sync_frequency": "60", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_list_bucket_min_readahead": "1000", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_list_buckets_max_chunk": "1000", 2026-02-20T11:59:01.972 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_log_http_headers": "", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_log_nonexistent_bucket": "false", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_log_object_name_utc": "false", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lua_enable": "true", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lua_max_memory_per_state": "128000", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_lua_max_runtime_per_state": "1000", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_attr_name_len": "0", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_attr_size": "0", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_attrs_num_in_req": "0", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_chunk_size": "4194304", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_concurrent_requests": "1024", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_control_aio": "8", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_dynamic_shards": "1999", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_listing_results": "5000", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_notify_retries": "10", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_objs_per_shard": "100000", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_put_param_size": "1048576", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_put_size": "5368709120", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_max_slo_entries": "1000", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_md_log_max_shards": "64", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_md_notify_interval_msec": "200", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_meta_sync_poll_interval": "20", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_meta_sync_spawn_window": "20", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_mime_types_file": "/etc/mime.types", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_mp_lock_max_time": "600", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_multi_obj_del_max_aio": "16", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_multipart_min_part_size": "5242880", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_multipart_part_upload_limit": "10000", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_fhcache_partitions": "3", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_fhcache_size": "2017", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_frontends": "rgw-nfs", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_lru_lanes": "5", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_max_gc": "300", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_namespace_expire_secs": "300", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_run_gc_threads": "false", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_run_lc_threads": "false", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_run_quota_threads": "false", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_run_restore_threads": "false", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_run_sync_thread": "false", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_s3_fast_attrs": "false", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_nfs_write_completion_interval_s": "10", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_num_async_rados_threads": "32", 2026-02-20T11:59:01.973 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_num_control_oids": "8", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_numa_node": "-1", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_obj_stripe_size": "4194304", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_obj_tombstone_cache_size": "1000", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_objexp_chunk_size": "100", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_objexp_gc_interval": "600", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_objexp_hints_num_shards": "127", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_olh_pending_timeout_sec": "3600", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_op_thread_suicide_timeout": "0", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_op_thread_timeout": "600", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_op_tracing": "false", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_opa_token": "", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_opa_url": "", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_opa_verify_ssl": "true", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ops_log_data_backlog": "5242880", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ops_log_rados": "false", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ops_log_socket_path": "", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_override_bucket_index_max_shards": "0", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_parquet_buffer_size": "16777216", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_period_push_interval": "2.000000", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_period_push_interval_max": "30.000000", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_period_root_pool": ".rgw.root", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_policy_reject_invalid_principals": "true", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_posix_cache_lanes": "3", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_posix_cache_lmdb_count": "3", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_posix_cache_max_buckets": "100", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_posix_cache_partitions": "3", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_posix_userdb_dir": "/var/lib/ceph/radosgw", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_print_continue": "true", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_print_prohibited_content_length": "false", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_put_obj_max_window_size": "67108864", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_put_obj_min_window_size": "16777216", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_rados_pool_recovery_priority": "5", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_rados_tracing": "false", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_ratelimit_interval": "60", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_realm": "", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_realm_id": "", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_realm_root_pool": ".rgw.root", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_redis_connection_pool_size": "512", 2026-02-20T11:59:01.974 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_region": "", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_region_root_pool": ".rgw.root", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_relaxed_region_enforcement": "false", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_relaxed_s3_bucket_names": "false", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_request_uri": "", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_reshard_batch_size": "64", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_reshard_bucket_lock_duration": "360", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_reshard_debug_interval": "-1", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_reshard_max_aio": "128", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_reshard_num_logs": "16", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_reshard_progress_judge_interval": "120", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_reshard_thread_interval": "600", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_reshardlog_threshold": "30000", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_resolve_cname": "false", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_rest_getusage_op_compat": "false", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_restore_debug_interval": "-1", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_restore_lock_max_time": "90", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_restore_max_objs": "32", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_restore_processor_period": "900", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_run_sync_thread": "true", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_s3_auth_disable_signature_url": "false", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_s3_auth_order": "sts, external, local", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_s3_auth_use_keystone": "false", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_s3_auth_use_ldap": "false", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_s3_auth_use_rados": "true", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_s3_auth_use_sts": "false", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_s3_client_max_sig_ver": "-1", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_s3_success_create_obj_status": "0", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_safe_max_objects_per_shard": "102400", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_scheduler_type": "throttler", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_script_uri": "", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_service_provider_name": "", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_shard_warning_threshold": "90.000000", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sts_client_id": "", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sts_client_secret": "", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sts_entry": "sts", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sts_key": "", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sts_max_session_duration": "43200", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sts_min_session_duration": "900", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sts_token_introspection_url": "", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_account_in_url": "false", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_auth_entry": "auth", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_auth_url": "", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_custom_header": "", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_enforce_content_length": "false", 2026-02-20T11:59:01.975 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_need_stats": "true", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_tenant_name": "", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_token_expiration": "86400", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_url": "", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_url_prefix": "swift", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_swift_versioning_enabled": "false", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_lease_period": "120", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_log_trim_interval": "1200", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_log_trim_max_buckets": "16", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_obj_etag_verify": "false", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_trace_history_size": "4096", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_trace_per_node_log_size": "32", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_thread_pool_size": "512", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_topic_persistency_max_retries": "0", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_topic_persistency_sleep_duration": "0", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_topic_persistency_time_to_live": "0", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_topic_require_publish_policy": "false", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_torrent_comment": "", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_torrent_createby": "", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_torrent_encoding": "", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_torrent_flag": "false", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_torrent_max_size": "5368709120", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_torrent_origin": "", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_torrent_sha_unit": "524288", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_torrent_tracker": "", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_trust_forwarded_https": "false", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_usage_log_flush_threshold": "1024", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_usage_log_key_transition": "true", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_usage_log_tick_interval": "30", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_usage_max_shards": "32", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_usage_max_user_shards": "1", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_use_opa_authz": "false", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_counters_cache": "false", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_counters_cache_size": "10000", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_default_quota_max_objects": "-1", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_default_quota_max_size": "-1", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_max_buckets": "1000", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_policies_max_num": "100", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_quota_sync_idle_users": "false", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_quota_sync_interval": "86400", 2026-02-20T11:59:01.976 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_quota_sync_wait_time": "86400", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_user_unique_email": "true", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_verify_ssl": "true", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_website_routing_rules_max_num": "50", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_zone": "", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_zone_id": "", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_zone_root_pool": ".rgw.root", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_zonegroup": "", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_zonegroup_id": "", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgwlc_auto_session_clear": "true", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rgwlc_skip_bucket_step": "false", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_block_size": "4096", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_bloom_bits_per_key": "20", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_cache_row_ratio": "0.000000", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_cache_shard_bits": "4", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_cache_size": "536870912", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_cache_type": "binned_lru", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_cf_compact_on_deletion": "true", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_collect_compaction_stats": "false", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_collect_extended_stats": "false", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_collect_memory_stats": "false", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_delete_range_threshold": "1048576", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_index_type": "binary_search", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_log_to_ceph_log": "true", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_metadata_block_size": "4096", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_partition_filters": "false", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_perf": "false", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rotating_keys_bootstrap_timeout": "30", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "rotating_keys_renewal_timeout": "10", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "run_dir": "/var/run/ceph", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_block_create": "true", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_cachepin_2q_in_ratio": "0.500000", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_cachepin_2q_out_ratio": "0.500000", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_cachepin_size_pershard": "2147483648", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_cachepin_type": "LRU", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_cbjournal_size": "5368709120", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_cold_tier_generations": "3", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_data_delta_based_overwrite": "0", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_default_max_object_size": "16777216", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_default_object_metadata_reservation": "16777216", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_device_size": "53687091200", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_disable_end_to_end_data_protection": "true", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_full_integrity_check": "false", 2026-02-20T11:59:01.977 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_hot_tier_generations": "5", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_journal_batch_capacity": "16", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_journal_batch_flush_size": "16777216", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_journal_iodepth_limit": "5", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_main_device_type": "SSD", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_max_concurrent_transactions": "128", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_max_data_allocation_size": "0", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_segment_cleaner_gc_formula": "cost_benefit", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "seastore_segment_size": "67108864", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "service_unique_id": "", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "set_keepcaps": "false", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "setgroup": "", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "setuser": "", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "setuser_match_path": "", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "subv_metrics_window_interval": "30", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "target_max_misplaced_ratio": "0.050000", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "thp": "false", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "threadpool_default_timeout": "60", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "threadpool_empty_queue_max_wait": "2", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "throttler_perf_counter": "true", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "tmp_dir": "/tmp", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "uadk_compressor_enabled": "false", 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout: "uadk_wd_sync_ctx_num": "2" 2026-02-20T11:59:01.978 INFO:teuthology.orchestra.run.trial153.stdout:} 2026-02-20T11:59:01.979 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2026-02-20T11:59:02.053 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:02.047+0000 7fcd36a51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:02.056 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:02.052+0000 7fcd36a51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:02.066 INFO:teuthology.orchestra.run.trial153.stdout:{} 2026-02-20T11:59:02.066 INFO:teuthology.orchestra.run.trial153.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2026-02-20T11:59:02.072 DEBUG:teuthology.orchestra.run.trial183:> type ceph-objectstore-tool 2026-02-20T11:59:02.089 INFO:teuthology.orchestra.run.trial183.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-20T11:59:02.090 DEBUG:teuthology.orchestra.run.trial153:> type ceph-objectstore-tool 2026-02-20T11:59:02.105 INFO:teuthology.orchestra.run.trial153.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-20T11:59:02.105 INFO:teuthology.run_tasks:Running task rados... 2026-02-20T11:59:02.122 INFO:tasks.rados:Beginning rados... 2026-02-20T11:59:02.122 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2026-02-20T11:59:02.122 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-perezjos-crimson-only-19-02-2026-PR64439'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_max_pg_log_entries': 2, 'osd_min_pg_log_entries': 1, 'osd_pg_log_trim_min': 0, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore cachepin type': '2Q', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '0c3148ca19c827ca2a7a9b92c2554d99507fd1f4'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '445d5c9ba18a4040e70e25303230c36c77773acb'}} 2026-02-20T11:59:02.122 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2026-02-20T11:59:02.122 DEBUG:teuthology.run_tasks:Unwinding manager rados 2026-02-20T11:59:02.135 INFO:tasks.rados:joining rados 2026-02-20T11:59:02.135 INFO:tasks.thrashosds.thrasher:starting do_thrash 2026-02-20T11:59:02.135 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-20T11:59:02.136 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-20T11:59:02.136 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3] 2026-02-20T11:59:02.136 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-20T11:59:02.136 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T11:59:02.136 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2026-02-20T11:59:02.136 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:59:02.137 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2026-02-20T11:59:02.137 INFO:tasks.rados:starting run 0 out of 1 2026-02-20T11:59:02.137 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2026-02-20T11:59:02.137 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2026-02-20T11:59:02.195 INFO:tasks.ceph.osd.2:Stopped 2026-02-20T11:59:02.223 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:02.218+0000 7ff41bd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:02.227 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:02.217+0000 7f7f8ca8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:02.227 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:02.223+0000 7ff41bd0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:02.229 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:02.225+0000 7f7f8ca8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:02.573 INFO:teuthology.orchestra.run.trial153.stderr:pool 'unique_pool_0' created 2026-02-20T11:59:02.588 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-20T11:59:02.710 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:02.700+0000 7f6eb6a51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:02.713 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:02.708+0000 7f6eb6a51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:03.577 INFO:teuthology.orchestra.run.trial153.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-20T11:59:04.593 DEBUG:teuthology.orchestra.run.trial153:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2026-02-20T11:59:04.641 INFO:tasks.rados.rados.0.trial153.stdout:adding op weight read -> 100 2026-02-20T11:59:04.642 INFO:tasks.rados.rados.0.trial153.stdout:adding op weight write -> 50 2026-02-20T11:59:04.642 INFO:tasks.rados.rados.0.trial153.stdout:adding op weight delete -> 50 2026-02-20T11:59:04.642 INFO:tasks.rados.rados.0.trial153.stdout:adding op weight snap_create -> 50 2026-02-20T11:59:04.642 INFO:tasks.rados.rados.0.trial153.stdout:adding op weight snap_remove -> 50 2026-02-20T11:59:04.642 INFO:tasks.rados.rados.0.trial153.stdout:adding op weight write_excl -> 50 2026-02-20T11:59:04.642 INFO:tasks.rados.rados.0.trial153.stdout:ceph version 20.3.0-5357-g0c3148ca (0c3148ca19c827ca2a7a9b92c2554d99507fd1f4) tentacle (dev - Debug) 2026-02-20T11:59:04.642 INFO:tasks.rados.rados.0.trial153.stdout:Configuration: 2026-02-20T11:59:04.642 INFO:tasks.rados.rados.0.trial153.stdout: Number of operations: 4000 2026-02-20T11:59:04.642 INFO:tasks.rados.rados.0.trial153.stdout: Number of objects: 50 2026-02-20T11:59:04.643 INFO:tasks.rados.rados.0.trial153.stdout: Max in flight operations: 16 2026-02-20T11:59:04.643 INFO:tasks.rados.rados.0.trial153.stdout: Object size (in bytes): 4000000 2026-02-20T11:59:04.643 INFO:tasks.rados.rados.0.trial153.stdout: Write stride min: 400000 2026-02-20T11:59:04.643 INFO:tasks.rados.rados.0.trial153.stdout: Write stride max: 800000 2026-02-20T11:59:05.584 INFO:tasks.rados.rados.0.trial153.stdout:1: write initial oid 1 2026-02-20T11:59:05.584 INFO:tasks.rados.rados.0.trial153.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2026-02-20T11:59:05.587 INFO:tasks.rados.rados.0.trial153.stdout:1: writing trial15342161-1 from 416807 to 892056 tid 1 2026-02-20T11:59:05.592 INFO:tasks.rados.rados.0.trial153.stdout:1: writing trial15342161-1 from 1542129 to 2016807 tid 2 2026-02-20T11:59:05.592 INFO:tasks.rados.rados.0.trial153.stdout:2: write initial oid 2 2026-02-20T11:59:05.592 INFO:tasks.rados.rados.0.trial153.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2026-02-20T11:59:05.595 INFO:tasks.rados.rados.0.trial153.stdout:2: writing trial15342161-2 from 416807 to 892056 tid 1 2026-02-20T11:59:05.600 INFO:tasks.rados.rados.0.trial153.stdout:2: writing trial15342161-2 from 1542129 to 2016807 tid 2 2026-02-20T11:59:05.600 INFO:tasks.rados.rados.0.trial153.stdout:3: write initial oid 3 2026-02-20T11:59:05.600 INFO:tasks.rados.rados.0.trial153.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2026-02-20T11:59:05.604 INFO:tasks.rados.rados.0.trial153.stdout:3: writing trial15342161-3 from 433614 to 984112 tid 1 2026-02-20T11:59:05.608 INFO:tasks.rados.rados.0.trial153.stdout:3: writing trial15342161-3 from 1600611 to 2033614 tid 2 2026-02-20T11:59:05.608 INFO:tasks.rados.rados.0.trial153.stdout:4: write initial oid 4 2026-02-20T11:59:05.608 INFO:tasks.rados.rados.0.trial153.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2026-02-20T11:59:05.612 INFO:tasks.rados.rados.0.trial153.stdout:4: writing trial15342161-4 from 450421 to 1076168 tid 1 2026-02-20T11:59:05.616 INFO:tasks.rados.rados.0.trial153.stdout:4: writing trial15342161-4 from 1659093 to 2050421 tid 2 2026-02-20T11:59:05.616 INFO:tasks.rados.rados.0.trial153.stdout:5: write initial oid 5 2026-02-20T11:59:05.616 INFO:tasks.rados.rados.0.trial153.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2026-02-20T11:59:05.621 INFO:tasks.rados.rados.0.trial153.stdout:5: writing trial15342161-5 from 467228 to 1168224 tid 1 2026-02-20T11:59:05.624 INFO:tasks.rados.rados.0.trial153.stdout:5: writing trial15342161-5 from 1717575 to 2067228 tid 2 2026-02-20T11:59:05.625 INFO:tasks.rados.rados.0.trial153.stdout:6: write initial oid 6 2026-02-20T11:59:05.625 INFO:tasks.rados.rados.0.trial153.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2026-02-20T11:59:05.630 INFO:tasks.rados.rados.0.trial153.stdout:6: writing trial15342161-6 from 484035 to 1260280 tid 1 2026-02-20T11:59:05.633 INFO:tasks.rados.rados.0.trial153.stdout:6: writing trial15342161-6 from 2059704 to 2084035 tid 2 2026-02-20T11:59:05.633 INFO:tasks.rados.rados.0.trial153.stdout:7: write initial oid 7 2026-02-20T11:59:05.633 INFO:tasks.rados.rados.0.trial153.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2026-02-20T11:59:05.636 INFO:tasks.rados.rados.0.trial153.stdout:7: writing trial15342161-7 from 500842 to 952336 tid 1 2026-02-20T11:59:05.641 INFO:tasks.rados.rados.0.trial153.stdout:7: writing trial15342161-7 from 1718186 to 2100842 tid 2 2026-02-20T11:59:05.641 INFO:tasks.rados.rados.0.trial153.stdout:8: write initial oid 8 2026-02-20T11:59:05.641 INFO:tasks.rados.rados.0.trial153.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2026-02-20T11:59:05.645 INFO:tasks.rados.rados.0.trial153.stdout:8: writing trial15342161-8 from 517649 to 1044392 tid 1 2026-02-20T11:59:05.649 INFO:tasks.rados.rados.0.trial153.stdout:8: writing trial15342161-8 from 1776668 to 2117649 tid 2 2026-02-20T11:59:05.649 INFO:tasks.rados.rados.0.trial153.stdout:9: write initial oid 9 2026-02-20T11:59:05.649 INFO:tasks.rados.rados.0.trial153.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2026-02-20T11:59:05.654 INFO:tasks.rados.rados.0.trial153.stdout:9: writing trial15342161-9 from 534456 to 1252801 tid 1 2026-02-20T11:59:05.657 INFO:tasks.rados.rados.0.trial153.stdout:9: writing trial15342161-9 from 1951503 to 2134456 tid 2 2026-02-20T11:59:05.657 INFO:tasks.rados.rados.0.trial153.stdout:10: write initial oid 10 2026-02-20T11:59:05.658 INFO:tasks.rados.rados.0.trial153.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2026-02-20T11:59:05.663 INFO:tasks.rados.rados.0.trial153.stdout:10: writing trial15342161-10 from 551263 to 1344857 tid 1 2026-02-20T11:59:05.666 INFO:tasks.rados.rados.0.trial153.stdout:10: writing trial15342161-10 from 1893632 to 2151263 tid 2 2026-02-20T11:59:05.666 INFO:tasks.rados.rados.0.trial153.stdout:11: write initial oid 11 2026-02-20T11:59:05.666 INFO:tasks.rados.rados.0.trial153.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2026-02-20T11:59:05.670 INFO:tasks.rados.rados.0.trial153.stdout:11: writing trial15342161-11 from 568070 to 1036913 tid 1 2026-02-20T11:59:05.675 INFO:tasks.rados.rados.0.trial153.stdout:11: writing trial15342161-11 from 1552114 to 2168070 tid 2 2026-02-20T11:59:05.675 INFO:tasks.rados.rados.0.trial153.stdout:12: write initial oid 12 2026-02-20T11:59:05.675 INFO:tasks.rados.rados.0.trial153.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2026-02-20T11:59:05.679 INFO:tasks.rados.rados.0.trial153.stdout:12: writing trial15342161-12 from 584877 to 1128969 tid 1 2026-02-20T11:59:05.683 INFO:tasks.rados.rados.0.trial153.stdout:12: writing trial15342161-12 from 1610596 to 2172599 tid 2 2026-02-20T11:59:05.684 INFO:tasks.rados.rados.0.trial153.stdout:13: write initial oid 13 2026-02-20T11:59:05.684 INFO:tasks.rados.rados.0.trial153.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2026-02-20T11:59:05.688 INFO:tasks.rados.rados.0.trial153.stdout:13: writing trial15342161-13 from 601684 to 1221025 tid 1 2026-02-20T11:59:05.692 INFO:tasks.rados.rados.0.trial153.stdout:13: writing trial15342161-13 from 1669078 to 2201684 tid 2 2026-02-20T11:59:05.692 INFO:tasks.rados.rados.0.trial153.stdout:14: write initial oid 14 2026-02-20T11:59:05.692 INFO:tasks.rados.rados.0.trial153.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2026-02-20T11:59:05.697 INFO:tasks.rados.rados.0.trial153.stdout:14: writing trial15342161-14 from 618491 to 1313081 tid 1 2026-02-20T11:59:05.701 INFO:tasks.rados.rados.0.trial153.stdout:14: writing trial15342161-14 from 2011207 to 2218491 tid 2 2026-02-20T11:59:05.701 INFO:tasks.rados.rados.0.trial153.stdout:15: write initial oid 15 2026-02-20T11:59:05.701 INFO:tasks.rados.rados.0.trial153.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2026-02-20T11:59:05.706 INFO:tasks.rados.rados.0.trial153.stdout:15: writing trial15342161-15 from 635298 to 1405137 tid 1 2026-02-20T11:59:05.710 INFO:tasks.rados.rados.0.trial153.stdout:15: writing trial15342161-15 from 2069689 to 2235298 tid 2 2026-02-20T11:59:05.710 INFO:tasks.rados.rados.0.trial153.stdout:16: write initial oid 16 2026-02-20T11:59:05.710 INFO:tasks.rados.rados.0.trial153.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2026-02-20T11:59:05.714 INFO:tasks.rados.rados.0.trial153.stdout:16: writing trial15342161-16 from 652105 to 1097193 tid 1 2026-02-20T11:59:05.719 INFO:tasks.rados.rados.0.trial153.stdout:16: writing trial15342161-16 from 1728171 to 2181159 tid 2 2026-02-20T11:59:05.719 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:05.719 INFO:tasks.rados.rados.0.trial153.stdout:1: finishing write tid 1 to trial15342161-1 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:1: finishing write tid 2 to trial15342161-1 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:1: finishing write tid 3 to trial15342161-1 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:1: finishing write tid 4 to trial15342161-1 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:1: oid 1 updating version 0 to 1 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:1: oid 1 updating version 1 to 3 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:1: oid 1 version 3 is already newer than 3 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:2: finishing write tid 1 to trial15342161-2 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:2: finishing write tid 2 to trial15342161-2 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:6: finishing write tid 1 to trial15342161-6 2026-02-20T11:59:05.720 INFO:tasks.rados.rados.0.trial153.stdout:2: finishing write tid 3 to trial15342161-2 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:2: finishing write tid 4 to trial15342161-2 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:2: oid 2 updating version 0 to 6 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:2: oid 2 version 6 is already newer than 4 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:2: oid 2 version 6 is already newer than 6 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 2 v 6 (ObjNum 1 snap 0 seq_num 1) dirty exists 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:6: finishing write tid 2 to trial15342161-6 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:1: done (15 left) 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:2: done (14 left) 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:17: write initial oid 17 2026-02-20T11:59:05.721 INFO:tasks.rados.rados.0.trial153.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2026-02-20T11:59:05.723 INFO:tasks.rados.rados.0.trial153.stdout:17: writing trial15342161-17 from 668912 to 1305602 tid 1 2026-02-20T11:59:05.728 INFO:tasks.rados.rados.0.trial153.stdout:17: writing trial15342161-17 from 1903006 to 2268912 tid 2 2026-02-20T11:59:05.728 INFO:tasks.rados.rados.0.trial153.stdout:18: write initial oid 18 2026-02-20T11:59:05.728 INFO:tasks.rados.rados.0.trial153.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2026-02-20T11:59:05.733 INFO:tasks.rados.rados.0.trial153.stdout:18: writing trial15342161-18 from 685719 to 1397658 tid 1 2026-02-20T11:59:05.738 INFO:tasks.rados.rados.0.trial153.stdout:18: writing trial15342161-18 from 1845135 to 2285719 tid 2 2026-02-20T11:59:05.738 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:05.738 INFO:tasks.rados.rados.0.trial153.stdout:6: finishing write tid 3 to trial15342161-6 2026-02-20T11:59:05.738 INFO:tasks.rados.rados.0.trial153.stdout:6: finishing write tid 4 to trial15342161-6 2026-02-20T11:59:05.738 INFO:tasks.rados.rados.0.trial153.stdout:6: oid 6 updating version 0 to 1 2026-02-20T11:59:05.738 INFO:tasks.rados.rados.0.trial153.stdout:6: oid 6 updating version 1 to 3 2026-02-20T11:59:05.739 INFO:tasks.rados.rados.0.trial153.stdout:6: oid 6 version 3 is already newer than 3 2026-02-20T11:59:05.739 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2026-02-20T11:59:05.739 INFO:tasks.rados.rados.0.trial153.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2026-02-20T11:59:05.739 INFO:tasks.rados.rados.0.trial153.stdout:9: finishing write tid 1 to trial15342161-9 2026-02-20T11:59:05.739 INFO:tasks.rados.rados.0.trial153.stdout:6: done (15 left) 2026-02-20T11:59:05.739 INFO:tasks.rados.rados.0.trial153.stdout:19: write initial oid 19 2026-02-20T11:59:05.739 INFO:tasks.rados.rados.0.trial153.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2026-02-20T11:59:05.743 INFO:tasks.rados.rados.0.trial153.stdout:19: writing trial15342161-19 from 702526 to 1489714 tid 1 2026-02-20T11:59:05.746 INFO:tasks.rados.rados.0.trial153.stdout:19: writing trial15342161-19 from 1903617 to 2302526 tid 2 2026-02-20T11:59:05.747 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:05.747 INFO:tasks.rados.rados.0.trial153.stdout:11: finishing write tid 1 to trial15342161-11 2026-02-20T11:59:05.747 INFO:tasks.rados.rados.0.trial153.stdout:11: finishing write tid 2 to trial15342161-11 2026-02-20T11:59:05.747 INFO:tasks.rados.rados.0.trial153.stdout:11: finishing write tid 3 to trial15342161-11 2026-02-20T11:59:05.747 INFO:tasks.rados.rados.0.trial153.stdout:11: finishing write tid 4 to trial15342161-11 2026-02-20T11:59:05.747 INFO:tasks.rados.rados.0.trial153.stdout:11: oid 11 updating version 0 to 1 2026-02-20T11:59:05.747 INFO:tasks.rados.rados.0.trial153.stdout:11: oid 11 updating version 1 to 3 2026-02-20T11:59:05.747 INFO:tasks.rados.rados.0.trial153.stdout:11: oid 11 version 3 is already newer than 3 2026-02-20T11:59:05.747 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2026-02-20T11:59:05.747 INFO:tasks.rados.rados.0.trial153.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2026-02-20T11:59:05.748 INFO:tasks.rados.rados.0.trial153.stdout:9: finishing write tid 2 to trial15342161-9 2026-02-20T11:59:05.748 INFO:tasks.rados.rados.0.trial153.stdout:14: finishing write tid 1 to trial15342161-14 2026-02-20T11:59:05.748 INFO:tasks.rados.rados.0.trial153.stdout:9: finishing write tid 3 to trial15342161-9 2026-02-20T11:59:05.748 INFO:tasks.rados.rados.0.trial153.stdout:11: done (15 left) 2026-02-20T11:59:05.748 INFO:tasks.rados.rados.0.trial153.stdout:20: write initial oid 20 2026-02-20T11:59:05.748 INFO:tasks.rados.rados.0.trial153.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2026-02-20T11:59:05.750 INFO:tasks.rados.rados.0.trial153.stdout:20: writing trial15342161-20 from 719333 to 1181770 tid 1 2026-02-20T11:59:05.755 INFO:tasks.rados.rados.0.trial153.stdout:20: writing trial15342161-20 from 1962099 to 2319333 tid 2 2026-02-20T11:59:05.755 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:05.755 INFO:tasks.rados.rados.0.trial153.stdout:9: finishing write tid 4 to trial15342161-9 2026-02-20T11:59:05.755 INFO:tasks.rados.rados.0.trial153.stdout:9: oid 9 updating version 0 to 9 2026-02-20T11:59:05.756 INFO:tasks.rados.rados.0.trial153.stdout:9: oid 9 version 9 is already newer than 7 2026-02-20T11:59:05.756 INFO:tasks.rados.rados.0.trial153.stdout:9: oid 9 version 9 is already newer than 9 2026-02-20T11:59:05.756 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 9 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2026-02-20T11:59:05.756 INFO:tasks.rados.rados.0.trial153.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2026-02-20T11:59:05.756 INFO:tasks.rados.rados.0.trial153.stdout:13: finishing write tid 1 to trial15342161-13 2026-02-20T11:59:05.756 INFO:tasks.rados.rados.0.trial153.stdout:9: done (15 left) 2026-02-20T11:59:05.756 INFO:tasks.rados.rados.0.trial153.stdout:21: write initial oid 21 2026-02-20T11:59:05.756 INFO:tasks.rados.rados.0.trial153.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2026-02-20T11:59:05.759 INFO:tasks.rados.rados.0.trial153.stdout:21: writing trial15342161-21 from 736140 to 1273826 tid 1 2026-02-20T11:59:05.764 INFO:tasks.rados.rados.0.trial153.stdout:21: writing trial15342161-21 from 2020581 to 2336140 tid 2 2026-02-20T11:59:05.764 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:05.764 INFO:tasks.rados.rados.0.trial153.stdout:14: finishing write tid 2 to trial15342161-14 2026-02-20T11:59:05.764 INFO:tasks.rados.rados.0.trial153.stdout:14: finishing write tid 3 to trial15342161-14 2026-02-20T11:59:05.764 INFO:tasks.rados.rados.0.trial153.stdout:14: finishing write tid 4 to trial15342161-14 2026-02-20T11:59:05.764 INFO:tasks.rados.rados.0.trial153.stdout:14: oid 14 updating version 0 to 4 2026-02-20T11:59:05.764 INFO:tasks.rados.rados.0.trial153.stdout:14: oid 14 updating version 4 to 6 2026-02-20T11:59:05.764 INFO:tasks.rados.rados.0.trial153.stdout:14: oid 14 version 6 is already newer than 6 2026-02-20T11:59:05.765 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2026-02-20T11:59:05.765 INFO:tasks.rados.rados.0.trial153.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2026-02-20T11:59:05.765 INFO:tasks.rados.rados.0.trial153.stdout:18: finishing write tid 1 to trial15342161-18 2026-02-20T11:59:05.765 INFO:tasks.rados.rados.0.trial153.stdout:15: finishing write tid 1 to trial15342161-15 2026-02-20T11:59:05.765 INFO:tasks.rados.rados.0.trial153.stdout:17: finishing write tid 1 to trial15342161-17 2026-02-20T11:59:05.765 INFO:tasks.rados.rados.0.trial153.stdout:14: done (15 left) 2026-02-20T11:59:05.765 INFO:tasks.rados.rados.0.trial153.stdout:22: write initial oid 22 2026-02-20T11:59:05.765 INFO:tasks.rados.rados.0.trial153.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2026-02-20T11:59:05.769 INFO:tasks.rados.rados.0.trial153.stdout:22: writing trial15342161-22 from 752947 to 1365882 tid 1 2026-02-20T11:59:05.773 INFO:tasks.rados.rados.0.trial153.stdout:22: writing trial15342161-22 from 1962710 to 2352947 tid 2 2026-02-20T11:59:05.773 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:05.773 INFO:tasks.rados.rados.0.trial153.stdout:13: finishing write tid 2 to trial15342161-13 2026-02-20T11:59:05.775 INFO:tasks.rados.rados.0.trial153.stdout:15: finishing write tid 2 to trial15342161-15 2026-02-20T11:59:05.775 INFO:tasks.rados.rados.0.trial153.stdout:18: finishing write tid 2 to trial15342161-18 2026-02-20T11:59:05.776 INFO:tasks.rados.rados.0.trial153.stdout:17: finishing write tid 2 to trial15342161-17 2026-02-20T11:59:05.777 INFO:tasks.rados.rados.0.trial153.stdout:18: finishing write tid 3 to trial15342161-18 2026-02-20T11:59:05.779 INFO:tasks.rados.rados.0.trial153.stdout:17: finishing write tid 3 to trial15342161-17 2026-02-20T11:59:05.781 INFO:tasks.rados.rados.0.trial153.stdout:17: finishing write tid 4 to trial15342161-17 2026-02-20T11:59:05.781 INFO:tasks.rados.rados.0.trial153.stdout:17: oid 17 updating version 0 to 3 2026-02-20T11:59:05.781 INFO:tasks.rados.rados.0.trial153.stdout:17: oid 17 version 3 is already newer than 3 2026-02-20T11:59:05.781 INFO:tasks.rados.rados.0.trial153.stdout:17: oid 17 version 3 is already newer than 1 2026-02-20T11:59:05.781 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2026-02-20T11:59:05.781 INFO:tasks.rados.rados.0.trial153.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2026-02-20T11:59:05.781 INFO:tasks.rados.rados.0.trial153.stdout:17: done (15 left) 2026-02-20T11:59:05.782 INFO:tasks.rados.rados.0.trial153.stdout:23: write initial oid 23 2026-02-20T11:59:05.782 INFO:tasks.rados.rados.0.trial153.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2026-02-20T11:59:05.787 INFO:tasks.rados.rados.0.trial153.stdout:23: writing trial15342161-23 from 769754 to 1457938 tid 1 2026-02-20T11:59:05.791 INFO:tasks.rados.rados.0.trial153.stdout:23: writing trial15342161-23 from 2021192 to 2369754 tid 2 2026-02-20T11:59:05.791 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:05.791 INFO:tasks.rados.rados.0.trial153.stdout:18: finishing write tid 4 to trial15342161-18 2026-02-20T11:59:05.791 INFO:tasks.rados.rados.0.trial153.stdout:18: oid 18 updating version 0 to 1 2026-02-20T11:59:05.791 INFO:tasks.rados.rados.0.trial153.stdout:18: oid 18 updating version 1 to 3 2026-02-20T11:59:05.792 INFO:tasks.rados.rados.0.trial153.stdout:18: oid 18 version 3 is already newer than 3 2026-02-20T11:59:05.792 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2026-02-20T11:59:05.792 INFO:tasks.rados.rados.0.trial153.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2026-02-20T11:59:05.792 INFO:tasks.rados.rados.0.trial153.stdout:15: finishing write tid 3 to trial15342161-15 2026-02-20T11:59:05.792 INFO:tasks.rados.rados.0.trial153.stdout:13: finishing write tid 3 to trial15342161-13 2026-02-20T11:59:05.792 INFO:tasks.rados.rados.0.trial153.stdout:18: done (15 left) 2026-02-20T11:59:05.792 INFO:tasks.rados.rados.0.trial153.stdout:24: write initial oid 24 2026-02-20T11:59:05.792 INFO:tasks.rados.rados.0.trial153.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2026-02-20T11:59:05.796 INFO:tasks.rados.rados.0.trial153.stdout:24: writing trial15342161-24 from 786561 to 1266347 tid 1 2026-02-20T11:59:05.801 INFO:tasks.rados.rados.0.trial153.stdout:24: writing trial15342161-24 from 1796027 to 2263691 tid 2 2026-02-20T11:59:05.801 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:05.801 INFO:tasks.rados.rados.0.trial153.stdout:13: finishing write tid 4 to trial15342161-13 2026-02-20T11:59:05.801 INFO:tasks.rados.rados.0.trial153.stdout:13: oid 13 updating version 0 to 12 2026-02-20T11:59:05.801 INFO:tasks.rados.rados.0.trial153.stdout:13: oid 13 version 12 is already newer than 10 2026-02-20T11:59:05.801 INFO:tasks.rados.rados.0.trial153.stdout:13: oid 13 version 12 is already newer than 12 2026-02-20T11:59:05.801 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 13 v 12 (ObjNum 12 snap 0 seq_num 12) dirty exists 2026-02-20T11:59:05.801 INFO:tasks.rados.rados.0.trial153.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2026-02-20T11:59:05.801 INFO:tasks.rados.rados.0.trial153.stdout:15: finishing write tid 4 to trial15342161-15 2026-02-20T11:59:05.801 INFO:tasks.rados.rados.0.trial153.stdout:15: oid 15 updating version 0 to 7 2026-02-20T11:59:05.802 INFO:tasks.rados.rados.0.trial153.stdout:15: oid 15 updating version 7 to 9 2026-02-20T11:59:05.802 INFO:tasks.rados.rados.0.trial153.stdout:15: oid 15 version 9 is already newer than 9 2026-02-20T11:59:05.802 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2026-02-20T11:59:05.802 INFO:tasks.rados.rados.0.trial153.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2026-02-20T11:59:05.802 INFO:tasks.rados.rados.0.trial153.stdout:19: finishing write tid 1 to trial15342161-19 2026-02-20T11:59:05.802 INFO:tasks.rados.rados.0.trial153.stdout:13: done (15 left) 2026-02-20T11:59:05.802 INFO:tasks.rados.rados.0.trial153.stdout:15: done (14 left) 2026-02-20T11:59:05.802 INFO:tasks.rados.rados.0.trial153.stdout:25: write initial oid 25 2026-02-20T11:59:05.802 INFO:tasks.rados.rados.0.trial153.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2026-02-20T11:59:05.804 INFO:tasks.rados.rados.0.trial153.stdout:25: writing trial15342161-25 from 403368 to 958403 tid 1 2026-02-20T11:59:05.808 INFO:tasks.rados.rados.0.trial153.stdout:25: writing trial15342161-25 from 1454509 to 2182184 tid 2 2026-02-20T11:59:05.809 INFO:tasks.rados.rados.0.trial153.stdout:26: write initial oid 26 2026-02-20T11:59:05.809 INFO:tasks.rados.rados.0.trial153.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2026-02-20T11:59:05.813 INFO:tasks.rados.rados.0.trial153.stdout:26: writing trial15342161-26 from 420175 to 1050459 tid 1 2026-02-20T11:59:05.818 INFO:tasks.rados.rados.0.trial153.stdout:26: writing trial15342161-26 from 1796638 to 2267971 tid 2 2026-02-20T11:59:05.818 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:05.818 INFO:tasks.rados.rados.0.trial153.stdout:19: finishing write tid 2 to trial15342161-19 2026-02-20T11:59:05.818 INFO:tasks.rados.rados.0.trial153.stdout:19: finishing write tid 3 to trial15342161-19 2026-02-20T11:59:05.818 INFO:tasks.rados.rados.0.trial153.stdout:19: finishing write tid 4 to trial15342161-19 2026-02-20T11:59:05.818 INFO:tasks.rados.rados.0.trial153.stdout:19: oid 19 updating version 0 to 12 2026-02-20T11:59:05.818 INFO:tasks.rados.rados.0.trial153.stdout:19: oid 19 version 12 is already newer than 12 2026-02-20T11:59:05.818 INFO:tasks.rados.rados.0.trial153.stdout:19: oid 19 version 12 is already newer than 10 2026-02-20T11:59:05.819 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 12 (ObjNum 18 snap 0 seq_num 18) dirty exists 2026-02-20T11:59:05.819 INFO:tasks.rados.rados.0.trial153.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2026-02-20T11:59:05.819 INFO:tasks.rados.rados.0.trial153.stdout:19: done (15 left) 2026-02-20T11:59:05.819 INFO:tasks.rados.rados.0.trial153.stdout:27: write initial oid 27 2026-02-20T11:59:05.819 INFO:tasks.rados.rados.0.trial153.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2026-02-20T11:59:05.822 INFO:tasks.rados.rados.0.trial153.stdout:27: writing trial15342161-27 from 436982 to 1142515 tid 1 2026-02-20T11:59:05.827 INFO:tasks.rados.rados.0.trial153.stdout:27: writing trial15342161-27 from 1855120 to 2436982 tid 2 2026-02-20T11:59:05.827 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:05.827 INFO:tasks.rados.rados.0.trial153.stdout:26: finishing write tid 1 to trial15342161-26 2026-02-20T11:59:05.837 INFO:tasks.rados.rados.0.trial153.stdout:26: finishing write tid 2 to trial15342161-26 2026-02-20T11:59:05.841 INFO:tasks.rados.rados.0.trial153.stdout:26: finishing write tid 3 to trial15342161-26 2026-02-20T11:59:05.843 INFO:tasks.rados.rados.0.trial153.stdout:26: finishing write tid 4 to trial15342161-26 2026-02-20T11:59:05.843 INFO:tasks.rados.rados.0.trial153.stdout:26: oid 26 updating version 0 to 4 2026-02-20T11:59:05.843 INFO:tasks.rados.rados.0.trial153.stdout:26: oid 26 updating version 4 to 6 2026-02-20T11:59:05.843 INFO:tasks.rados.rados.0.trial153.stdout:26: oid 26 version 6 is already newer than 6 2026-02-20T11:59:05.843 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2026-02-20T11:59:05.843 INFO:tasks.rados.rados.0.trial153.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2026-02-20T11:59:05.843 INFO:tasks.rados.rados.0.trial153.stdout:26: done (15 left) 2026-02-20T11:59:05.843 INFO:tasks.rados.rados.0.trial153.stdout:28: write initial oid 28 2026-02-20T11:59:05.843 INFO:tasks.rados.rados.0.trial153.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2026-02-20T11:59:05.847 INFO:tasks.rados.rados.0.trial153.stdout:28: writing trial15342161-28 from 453789 to 1234571 tid 1 2026-02-20T11:59:05.852 INFO:tasks.rados.rados.0.trial153.stdout:28: writing trial15342161-28 from 1913602 to 2388604 tid 2 2026-02-20T11:59:05.852 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:07.196 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3] 2026-02-20T11:59:07.196 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-20T11:59:07.197 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2026-02-20T11:59:07.197 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-20T11:59:07.197 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-20T11:59:07.219 INFO:tasks.ceph.osd.2:Started 2026-02-20T11:59:07.219 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:07.307 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:07.310 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:12.312 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:12.385 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:12.388 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:13.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2026-02-20T11:59:17.389 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:17.461 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:17.465 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:19.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~6s 2026-02-20T11:59:22.466 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:22.537 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:22.540 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:24.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2026-02-20T11:59:27.491 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:27.525 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:27.526 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:27.541 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:27.609 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:27.613 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:28.590 INFO:tasks.rados.rados.0.trial153.stdout:12: finishing write tid 1 to trial15342161-12 2026-02-20T11:59:28.592 INFO:tasks.rados.rados.0.trial153.stdout:7: finishing write tid 1 to trial15342161-7 2026-02-20T11:59:28.596 INFO:tasks.rados.rados.0.trial153.stdout:3: finishing write tid 1 to trial15342161-3 2026-02-20T11:59:28.598 INFO:tasks.rados.rados.0.trial153.stdout:24: finishing write tid 1 to trial15342161-24 2026-02-20T11:59:28.600 INFO:tasks.rados.rados.0.trial153.stdout:10: finishing write tid 1 to trial15342161-10 2026-02-20T11:59:28.609 INFO:tasks.rados.rados.0.trial153.stdout:27: finishing write tid 1 to trial15342161-27 2026-02-20T11:59:28.613 INFO:tasks.rados.rados.0.trial153.stdout:7: finishing write tid 2 to trial15342161-7 2026-02-20T11:59:28.614 INFO:tasks.rados.rados.0.trial153.stdout:5: finishing write tid 1 to trial15342161-5 2026-02-20T11:59:28.619 INFO:tasks.rados.rados.0.trial153.stdout:7: finishing write tid 3 to trial15342161-7 2026-02-20T11:59:28.621 INFO:tasks.rados.rados.0.trial153.stdout:3: finishing write tid 2 to trial15342161-3 2026-02-20T11:59:28.623 INFO:tasks.rados.rados.0.trial153.stdout:7: finishing write tid 4 to trial15342161-7 2026-02-20T11:59:28.623 INFO:tasks.rados.rados.0.trial153.stdout:7: oid 7 updating version 0 to 1 2026-02-20T11:59:28.623 INFO:tasks.rados.rados.0.trial153.stdout:7: oid 7 updating version 1 to 3 2026-02-20T11:59:28.623 INFO:tasks.rados.rados.0.trial153.stdout:7: oid 7 version 3 is already newer than 3 2026-02-20T11:59:28.623 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2026-02-20T11:59:28.623 INFO:tasks.rados.rados.0.trial153.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2026-02-20T11:59:28.623 INFO:tasks.rados.rados.0.trial153.stdout:7: done (15 left) 2026-02-20T11:59:28.623 INFO:tasks.rados.rados.0.trial153.stdout:29: write initial oid 29 2026-02-20T11:59:28.623 INFO:tasks.rados.rados.0.trial153.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2026-02-20T11:59:28.626 INFO:tasks.rados.rados.0.trial153.stdout:29: writing trial15342161-29 from 470596 to 926627 tid 1 2026-02-20T11:59:28.631 INFO:tasks.rados.rados.0.trial153.stdout:29: writing trial15342161-29 from 1572084 to 2190744 tid 2 2026-02-20T11:59:28.631 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.631 INFO:tasks.rados.rados.0.trial153.stdout:3: finishing write tid 3 to trial15342161-3 2026-02-20T11:59:28.632 INFO:tasks.rados.rados.0.trial153.stdout:3: finishing write tid 4 to trial15342161-3 2026-02-20T11:59:28.632 INFO:tasks.rados.rados.0.trial153.stdout:3: oid 3 updating version 0 to 1 2026-02-20T11:59:28.632 INFO:tasks.rados.rados.0.trial153.stdout:3: oid 3 updating version 1 to 3 2026-02-20T11:59:28.632 INFO:tasks.rados.rados.0.trial153.stdout:3: oid 3 version 3 is already newer than 3 2026-02-20T11:59:28.633 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2026-02-20T11:59:28.633 INFO:tasks.rados.rados.0.trial153.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2026-02-20T11:59:28.633 INFO:tasks.rados.rados.0.trial153.stdout:3: done (15 left) 2026-02-20T11:59:28.633 INFO:tasks.rados.rados.0.trial153.stdout:30: write initial oid 30 2026-02-20T11:59:28.633 INFO:tasks.rados.rados.0.trial153.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2026-02-20T11:59:28.635 INFO:tasks.rados.rados.0.trial153.stdout:30: writing trial15342161-30 from 487403 to 1018683 tid 1 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:30: writing trial15342161-30 from 1514213 to 1992884 tid 2 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:5: finishing write tid 2 to trial15342161-5 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:24: finishing write tid 2 to trial15342161-24 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:5: finishing write tid 3 to trial15342161-5 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:5: finishing write tid 4 to trial15342161-5 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:5: oid 5 updating version 0 to 3 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:5: oid 5 version 3 is already newer than 3 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:5: oid 5 version 3 is already newer than 1 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:27: finishing write tid 2 to trial15342161-27 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:12: finishing write tid 2 to trial15342161-12 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:5: done (15 left) 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:31: write initial oid 31 2026-02-20T11:59:28.640 INFO:tasks.rados.rados.0.trial153.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2026-02-20T11:59:28.643 INFO:tasks.rados.rados.0.trial153.stdout:31: writing trial15342161-31 from 504210 to 1110739 tid 1 2026-02-20T11:59:28.647 INFO:tasks.rados.rados.0.trial153.stdout:31: writing trial15342161-31 from 1572695 to 2195024 tid 2 2026-02-20T11:59:28.647 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.647 INFO:tasks.rados.rados.0.trial153.stdout:27: finishing write tid 3 to trial15342161-27 2026-02-20T11:59:28.647 INFO:tasks.rados.rados.0.trial153.stdout:10: finishing write tid 2 to trial15342161-10 2026-02-20T11:59:28.647 INFO:tasks.rados.rados.0.trial153.stdout:4: finishing write tid 1 to trial15342161-4 2026-02-20T11:59:28.647 INFO:tasks.rados.rados.0.trial153.stdout:24: finishing write tid 3 to trial15342161-24 2026-02-20T11:59:28.647 INFO:tasks.rados.rados.0.trial153.stdout:27: finishing write tid 4 to trial15342161-27 2026-02-20T11:59:28.647 INFO:tasks.rados.rados.0.trial153.stdout:27: oid 27 updating version 0 to 1 2026-02-20T11:59:28.647 INFO:tasks.rados.rados.0.trial153.stdout:27: oid 27 updating version 1 to 3 2026-02-20T11:59:28.647 INFO:tasks.rados.rados.0.trial153.stdout:27: oid 27 version 3 is already newer than 3 2026-02-20T11:59:28.648 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 27 v 3 (ObjNum 26 snap 0 seq_num 26) dirty exists 2026-02-20T11:59:28.648 INFO:tasks.rados.rados.0.trial153.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2026-02-20T11:59:28.648 INFO:tasks.rados.rados.0.trial153.stdout:12: finishing write tid 3 to trial15342161-12 2026-02-20T11:59:28.648 INFO:tasks.rados.rados.0.trial153.stdout:27: done (15 left) 2026-02-20T11:59:28.648 INFO:tasks.rados.rados.0.trial153.stdout:32: write initial oid 32 2026-02-20T11:59:28.648 INFO:tasks.rados.rados.0.trial153.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2026-02-20T11:59:28.652 INFO:tasks.rados.rados.0.trial153.stdout:32: writing trial15342161-32 from 521017 to 1319148 tid 1 2026-02-20T11:59:28.656 INFO:tasks.rados.rados.0.trial153.stdout:32: writing trial15342161-32 from 1747530 to 2229870 tid 2 2026-02-20T11:59:28.656 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.656 INFO:tasks.rados.rados.0.trial153.stdout:10: finishing write tid 3 to trial15342161-10 2026-02-20T11:59:28.657 INFO:tasks.rados.rados.0.trial153.stdout:24: finishing write tid 4 to trial15342161-24 2026-02-20T11:59:28.657 INFO:tasks.rados.rados.0.trial153.stdout:24: oid 24 updating version 0 to 1 2026-02-20T11:59:28.657 INFO:tasks.rados.rados.0.trial153.stdout:24: oid 24 updating version 1 to 3 2026-02-20T11:59:28.657 INFO:tasks.rados.rados.0.trial153.stdout:24: oid 24 version 3 is already newer than 3 2026-02-20T11:59:28.657 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 24 v 3 (ObjNum 23 snap 0 seq_num 23) dirty exists 2026-02-20T11:59:28.657 INFO:tasks.rados.rados.0.trial153.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2026-02-20T11:59:28.657 INFO:tasks.rados.rados.0.trial153.stdout:8: finishing write tid 1 to trial15342161-8 2026-02-20T11:59:28.657 INFO:tasks.rados.rados.0.trial153.stdout:24: done (15 left) 2026-02-20T11:59:28.657 INFO:tasks.rados.rados.0.trial153.stdout:33: write initial oid 33 2026-02-20T11:59:28.657 INFO:tasks.rados.rados.0.trial153.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2026-02-20T11:59:28.660 INFO:tasks.rados.rados.0.trial153.stdout:33: writing trial15342161-33 from 537824 to 1011204 tid 1 2026-02-20T11:59:28.666 INFO:tasks.rados.rados.0.trial153.stdout:33: writing trial15342161-33 from 1806012 to 2432010 tid 2 2026-02-20T11:59:28.666 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.666 INFO:tasks.rados.rados.0.trial153.stdout:12: finishing write tid 4 to trial15342161-12 2026-02-20T11:59:28.666 INFO:tasks.rados.rados.0.trial153.stdout:12: oid 12 updating version 0 to 1 2026-02-20T11:59:28.666 INFO:tasks.rados.rados.0.trial153.stdout:12: oid 12 updating version 1 to 3 2026-02-20T11:59:28.666 INFO:tasks.rados.rados.0.trial153.stdout:12: oid 12 version 3 is already newer than 3 2026-02-20T11:59:28.666 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2026-02-20T11:59:28.666 INFO:tasks.rados.rados.0.trial153.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2026-02-20T11:59:28.666 INFO:tasks.rados.rados.0.trial153.stdout:22: finishing write tid 1 to trial15342161-22 2026-02-20T11:59:28.666 INFO:tasks.rados.rados.0.trial153.stdout:10: finishing write tid 4 to trial15342161-10 2026-02-20T11:59:28.667 INFO:tasks.rados.rados.0.trial153.stdout:10: oid 10 updating version 0 to 1 2026-02-20T11:59:28.667 INFO:tasks.rados.rados.0.trial153.stdout:10: oid 10 updating version 1 to 3 2026-02-20T11:59:28.667 INFO:tasks.rados.rados.0.trial153.stdout:10: oid 10 version 3 is already newer than 3 2026-02-20T11:59:28.667 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2026-02-20T11:59:28.667 INFO:tasks.rados.rados.0.trial153.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2026-02-20T11:59:28.667 INFO:tasks.rados.rados.0.trial153.stdout:4: finishing write tid 2 to trial15342161-4 2026-02-20T11:59:28.667 INFO:tasks.rados.rados.0.trial153.stdout:10: done (15 left) 2026-02-20T11:59:28.667 INFO:tasks.rados.rados.0.trial153.stdout:12: done (14 left) 2026-02-20T11:59:28.667 INFO:tasks.rados.rados.0.trial153.stdout:34: write initial oid 34 2026-02-20T11:59:28.667 INFO:tasks.rados.rados.0.trial153.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2026-02-20T11:59:28.669 INFO:tasks.rados.rados.0.trial153.stdout:34: writing trial15342161-34 from 554631 to 1103260 tid 1 2026-02-20T11:59:28.674 INFO:tasks.rados.rados.0.trial153.stdout:34: writing trial15342161-34 from 1748141 to 2234150 tid 2 2026-02-20T11:59:28.674 INFO:tasks.rados.rados.0.trial153.stdout:35: write initial oid 35 2026-02-20T11:59:28.674 INFO:tasks.rados.rados.0.trial153.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2026-02-20T11:59:28.678 INFO:tasks.rados.rados.0.trial153.stdout:35: writing trial15342161-35 from 571438 to 1195316 tid 1 2026-02-20T11:59:28.683 INFO:tasks.rados.rados.0.trial153.stdout:35: writing trial15342161-35 from 1806623 to 2436290 tid 2 2026-02-20T11:59:28.683 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.683 INFO:tasks.rados.rados.0.trial153.stdout:8: finishing write tid 2 to trial15342161-8 2026-02-20T11:59:28.683 INFO:tasks.rados.rados.0.trial153.stdout:29: finishing write tid 1 to trial15342161-29 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:20: finishing write tid 1 to trial15342161-20 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:4: finishing write tid 3 to trial15342161-4 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:4: finishing write tid 4 to trial15342161-4 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:4: oid 4 updating version 0 to 6 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:4: oid 4 version 6 is already newer than 6 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:4: oid 4 version 6 is already newer than 4 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:22: finishing write tid 2 to trial15342161-22 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:8: finishing write tid 3 to trial15342161-8 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:8: finishing write tid 4 to trial15342161-8 2026-02-20T11:59:28.684 INFO:tasks.rados.rados.0.trial153.stdout:8: oid 8 updating version 0 to 4 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:8: oid 8 updating version 4 to 6 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:8: oid 8 version 6 is already newer than 6 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:16: finishing write tid 1 to trial15342161-16 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:32: finishing write tid 1 to trial15342161-32 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:29: finishing write tid 2 to trial15342161-29 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:22: finishing write tid 3 to trial15342161-22 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:22: finishing write tid 4 to trial15342161-22 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:22: oid 22 updating version 0 to 3 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:22: oid 22 version 3 is already newer than 3 2026-02-20T11:59:28.685 INFO:tasks.rados.rados.0.trial153.stdout:22: oid 22 version 3 is already newer than 1 2026-02-20T11:59:28.686 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2026-02-20T11:59:28.686 INFO:tasks.rados.rados.0.trial153.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2026-02-20T11:59:28.686 INFO:tasks.rados.rados.0.trial153.stdout:4: done (15 left) 2026-02-20T11:59:28.686 INFO:tasks.rados.rados.0.trial153.stdout:8: done (14 left) 2026-02-20T11:59:28.686 INFO:tasks.rados.rados.0.trial153.stdout:22: done (13 left) 2026-02-20T11:59:28.686 INFO:tasks.rados.rados.0.trial153.stdout:36: write initial oid 36 2026-02-20T11:59:28.686 INFO:tasks.rados.rados.0.trial153.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2026-02-20T11:59:28.688 INFO:tasks.rados.rados.0.trial153.stdout:36: writing trial15342161-36 from 588245 to 1287372 tid 1 2026-02-20T11:59:28.692 INFO:tasks.rados.rados.0.trial153.stdout:36: writing trial15342161-36 from 1865105 to 2354783 tid 2 2026-02-20T11:59:28.692 INFO:tasks.rados.rados.0.trial153.stdout:37: write initial oid 37 2026-02-20T11:59:28.692 INFO:tasks.rados.rados.0.trial153.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2026-02-20T11:59:28.697 INFO:tasks.rados.rados.0.trial153.stdout:37: writing trial15342161-37 from 605052 to 1379428 tid 1 2026-02-20T11:59:28.701 INFO:tasks.rados.rados.0.trial153.stdout:37: writing trial15342161-37 from 1923587 to 2556923 tid 2 2026-02-20T11:59:28.702 INFO:tasks.rados.rados.0.trial153.stdout:38: write initial oid 38 2026-02-20T11:59:28.702 INFO:tasks.rados.rados.0.trial153.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2026-02-20T11:59:28.705 INFO:tasks.rados.rados.0.trial153.stdout:38: writing trial15342161-38 from 621859 to 1071484 tid 1 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout:38: writing trial15342161-38 from 1865716 to 2621859 tid 2 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout:32: finishing write tid 2 to trial15342161-32 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout:29: finishing write tid 3 to trial15342161-29 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout:29: finishing write tid 4 to trial15342161-29 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout:29: oid 29 updating version 0 to 13 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout:29: oid 29 updating version 13 to 15 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout:29: oid 29 version 15 is already newer than 15 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 29 v 15 (ObjNum 28 snap 0 seq_num 28) dirty exists 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2026-02-20T11:59:28.712 INFO:tasks.rados.rados.0.trial153.stdout:23: finishing write tid 1 to trial15342161-23 2026-02-20T11:59:28.713 INFO:tasks.rados.rados.0.trial153.stdout:32: finishing write tid 3 to trial15342161-32 2026-02-20T11:59:28.713 INFO:tasks.rados.rados.0.trial153.stdout:29: done (15 left) 2026-02-20T11:59:28.713 INFO:tasks.rados.rados.0.trial153.stdout:39: write initial oid 39 2026-02-20T11:59:28.713 INFO:tasks.rados.rados.0.trial153.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2026-02-20T11:59:28.715 INFO:tasks.rados.rados.0.trial153.stdout:39: writing trial15342161-39 from 638666 to 1163540 tid 1 2026-02-20T11:59:28.721 INFO:tasks.rados.rados.0.trial153.stdout:39: writing trial15342161-39 from 1924198 to 2561203 tid 2 2026-02-20T11:59:28.721 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.721 INFO:tasks.rados.rados.0.trial153.stdout:32: finishing write tid 4 to trial15342161-32 2026-02-20T11:59:28.721 INFO:tasks.rados.rados.0.trial153.stdout:32: oid 32 updating version 0 to 7 2026-02-20T11:59:28.721 INFO:tasks.rados.rados.0.trial153.stdout:32: oid 32 updating version 7 to 9 2026-02-20T11:59:28.722 INFO:tasks.rados.rados.0.trial153.stdout:32: oid 32 version 9 is already newer than 9 2026-02-20T11:59:28.722 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2026-02-20T11:59:28.722 INFO:tasks.rados.rados.0.trial153.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2026-02-20T11:59:28.722 INFO:tasks.rados.rados.0.trial153.stdout:20: finishing write tid 2 to trial15342161-20 2026-02-20T11:59:28.722 INFO:tasks.rados.rados.0.trial153.stdout:32: done (15 left) 2026-02-20T11:59:28.722 INFO:tasks.rados.rados.0.trial153.stdout:40: write initial oid 40 2026-02-20T11:59:28.722 INFO:tasks.rados.rados.0.trial153.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2026-02-20T11:59:28.726 INFO:tasks.rados.rados.0.trial153.stdout:40: writing trial15342161-40 from 655473 to 1371949 tid 1 2026-02-20T11:59:28.731 INFO:tasks.rados.rados.0.trial153.stdout:40: writing trial15342161-40 from 2099033 to 2655473 tid 2 2026-02-20T11:59:28.731 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.731 INFO:tasks.rados.rados.0.trial153.stdout:20: finishing write tid 3 to trial15342161-20 2026-02-20T11:59:28.731 INFO:tasks.rados.rados.0.trial153.stdout:23: finishing write tid 2 to trial15342161-23 2026-02-20T11:59:28.732 INFO:tasks.rados.rados.0.trial153.stdout:20: finishing write tid 4 to trial15342161-20 2026-02-20T11:59:28.732 INFO:tasks.rados.rados.0.trial153.stdout:20: oid 20 updating version 0 to 1 2026-02-20T11:59:28.732 INFO:tasks.rados.rados.0.trial153.stdout:20: oid 20 updating version 1 to 3 2026-02-20T11:59:28.732 INFO:tasks.rados.rados.0.trial153.stdout:20: oid 20 version 3 is already newer than 3 2026-02-20T11:59:28.732 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2026-02-20T11:59:28.732 INFO:tasks.rados.rados.0.trial153.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2026-02-20T11:59:28.732 INFO:tasks.rados.rados.0.trial153.stdout:25: finishing write tid 1 to trial15342161-25 2026-02-20T11:59:28.732 INFO:tasks.rados.rados.0.trial153.stdout:20: done (15 left) 2026-02-20T11:59:28.732 INFO:tasks.rados.rados.0.trial153.stdout:41: write initial oid 41 2026-02-20T11:59:28.732 INFO:tasks.rados.rados.0.trial153.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2026-02-20T11:59:28.736 INFO:tasks.rados.rados.0.trial153.stdout:41: writing trial15342161-41 from 672280 to 1464005 tid 1 2026-02-20T11:59:28.741 INFO:tasks.rados.rados.0.trial153.stdout:41: writing trial15342161-41 from 2157515 to 2672280 tid 2 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:23: finishing write tid 3 to trial15342161-23 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:37: finishing write tid 1 to trial15342161-37 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:33: finishing write tid 1 to trial15342161-33 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:23: finishing write tid 4 to trial15342161-23 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:23: oid 23 updating version 0 to 6 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:23: oid 23 version 6 is already newer than 6 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:23: oid 23 version 6 is already newer than 4 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:38: finishing write tid 1 to trial15342161-38 2026-02-20T11:59:28.742 INFO:tasks.rados.rados.0.trial153.stdout:28: finishing write tid 1 to trial15342161-28 2026-02-20T11:59:28.743 INFO:tasks.rados.rados.0.trial153.stdout:23: done (15 left) 2026-02-20T11:59:28.743 INFO:tasks.rados.rados.0.trial153.stdout:42: write initial oid 42 2026-02-20T11:59:28.743 INFO:tasks.rados.rados.0.trial153.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2026-02-20T11:59:28.745 INFO:tasks.rados.rados.0.trial153.stdout:42: writing trial15342161-42 from 689087 to 1156061 tid 1 2026-02-20T11:59:28.751 INFO:tasks.rados.rados.0.trial153.stdout:42: writing trial15342161-42 from 1699644 to 2483976 tid 2 2026-02-20T11:59:28.751 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.751 INFO:tasks.rados.rados.0.trial153.stdout:37: finishing write tid 2 to trial15342161-37 2026-02-20T11:59:28.751 INFO:tasks.rados.rados.0.trial153.stdout:37: finishing write tid 3 to trial15342161-37 2026-02-20T11:59:28.751 INFO:tasks.rados.rados.0.trial153.stdout:38: finishing write tid 2 to trial15342161-38 2026-02-20T11:59:28.751 INFO:tasks.rados.rados.0.trial153.stdout:37: finishing write tid 4 to trial15342161-37 2026-02-20T11:59:28.752 INFO:tasks.rados.rados.0.trial153.stdout:37: oid 37 updating version 0 to 9 2026-02-20T11:59:28.752 INFO:tasks.rados.rados.0.trial153.stdout:37: oid 37 version 9 is already newer than 9 2026-02-20T11:59:28.752 INFO:tasks.rados.rados.0.trial153.stdout:37: oid 37 version 9 is already newer than 7 2026-02-20T11:59:28.752 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2026-02-20T11:59:28.752 INFO:tasks.rados.rados.0.trial153.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2026-02-20T11:59:28.752 INFO:tasks.rados.rados.0.trial153.stdout:38: finishing write tid 3 to trial15342161-38 2026-02-20T11:59:28.752 INFO:tasks.rados.rados.0.trial153.stdout:16: finishing write tid 2 to trial15342161-16 2026-02-20T11:59:28.752 INFO:tasks.rados.rados.0.trial153.stdout:38: finishing write tid 4 to trial15342161-38 2026-02-20T11:59:28.752 INFO:tasks.rados.rados.0.trial153.stdout:38: oid 38 updating version 0 to 10 2026-02-20T11:59:28.752 INFO:tasks.rados.rados.0.trial153.stdout:38: oid 38 updating version 10 to 12 2026-02-20T11:59:28.753 INFO:tasks.rados.rados.0.trial153.stdout:38: oid 38 version 12 is already newer than 12 2026-02-20T11:59:28.753 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2026-02-20T11:59:28.753 INFO:tasks.rados.rados.0.trial153.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2026-02-20T11:59:28.753 INFO:tasks.rados.rados.0.trial153.stdout:37: done (15 left) 2026-02-20T11:59:28.753 INFO:tasks.rados.rados.0.trial153.stdout:38: done (14 left) 2026-02-20T11:59:28.753 INFO:tasks.rados.rados.0.trial153.stdout:43: write initial oid 43 2026-02-20T11:59:28.753 INFO:tasks.rados.rados.0.trial153.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2026-02-20T11:59:28.755 INFO:tasks.rados.rados.0.trial153.stdout:43: writing trial15342161-43 from 705894 to 1248117 tid 1 2026-02-20T11:59:28.759 INFO:tasks.rados.rados.0.trial153.stdout:43: writing trial15342161-43 from 1758126 to 2402469 tid 2 2026-02-20T11:59:28.760 INFO:tasks.rados.rados.0.trial153.stdout:44: write initial oid 44 2026-02-20T11:59:28.760 INFO:tasks.rados.rados.0.trial153.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2026-02-20T11:59:28.764 INFO:tasks.rados.rados.0.trial153.stdout:44: writing trial15342161-44 from 722701 to 1340173 tid 1 2026-02-20T11:59:28.770 INFO:tasks.rados.rados.0.trial153.stdout:44: writing trial15342161-44 from 1816608 to 2604609 tid 2 2026-02-20T11:59:28.770 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.770 INFO:tasks.rados.rados.0.trial153.stdout:28: finishing write tid 2 to trial15342161-28 2026-02-20T11:59:28.770 INFO:tasks.rados.rados.0.trial153.stdout:16: finishing write tid 3 to trial15342161-16 2026-02-20T11:59:28.770 INFO:tasks.rados.rados.0.trial153.stdout:41: finishing write tid 1 to trial15342161-41 2026-02-20T11:59:28.770 INFO:tasks.rados.rados.0.trial153.stdout:28: finishing write tid 3 to trial15342161-28 2026-02-20T11:59:28.770 INFO:tasks.rados.rados.0.trial153.stdout:28: finishing write tid 4 to trial15342161-28 2026-02-20T11:59:28.770 INFO:tasks.rados.rados.0.trial153.stdout:28: oid 28 updating version 0 to 7 2026-02-20T11:59:28.770 INFO:tasks.rados.rados.0.trial153.stdout:28: oid 28 updating version 7 to 9 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:28: oid 28 version 9 is already newer than 9 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 28 v 9 (ObjNum 27 snap 0 seq_num 27) dirty exists 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:16: finishing write tid 4 to trial15342161-16 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:16: oid 16 updating version 0 to 6 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:16: oid 16 version 6 is already newer than 6 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:16: oid 16 version 6 is already newer than 4 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:25: finishing write tid 2 to trial15342161-25 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:35: finishing write tid 1 to trial15342161-35 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:16: done (15 left) 2026-02-20T11:59:28.771 INFO:tasks.rados.rados.0.trial153.stdout:28: done (14 left) 2026-02-20T11:59:28.772 INFO:tasks.rados.rados.0.trial153.stdout:45: write initial oid 45 2026-02-20T11:59:28.772 INFO:tasks.rados.rados.0.trial153.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2026-02-20T11:59:28.774 INFO:tasks.rados.rados.0.trial153.stdout:45: writing trial15342161-45 from 739508 to 1432229 tid 1 2026-02-20T11:59:28.779 INFO:tasks.rados.rados.0.trial153.stdout:45: writing trial15342161-45 from 1875090 to 2523102 tid 2 2026-02-20T11:59:28.779 INFO:tasks.rados.rados.0.trial153.stdout:46: write initial oid 46 2026-02-20T11:59:28.779 INFO:tasks.rados.rados.0.trial153.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2026-02-20T11:59:28.784 INFO:tasks.rados.rados.0.trial153.stdout:46: writing trial15342161-46 from 756315 to 1524285 tid 1 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout:46: writing trial15342161-46 from 1933572 to 2725242 tid 2 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout:41: finishing write tid 2 to trial15342161-41 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout:39: finishing write tid 1 to trial15342161-39 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout:43: finishing write tid 1 to trial15342161-43 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout:25: finishing write tid 3 to trial15342161-25 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout:41: finishing write tid 3 to trial15342161-41 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout:25: finishing write tid 4 to trial15342161-25 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout:25: oid 25 updating version 0 to 6 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout:25: oid 25 version 6 is already newer than 4 2026-02-20T11:59:28.790 INFO:tasks.rados.rados.0.trial153.stdout:25: oid 25 version 6 is already newer than 6 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:41: finishing write tid 4 to trial15342161-41 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:41: oid 41 updating version 0 to 6 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:41: oid 41 version 6 is already newer than 6 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:41: oid 41 version 6 is already newer than 4 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:35: finishing write tid 2 to trial15342161-35 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:25: done (15 left) 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:41: done (14 left) 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:47: write initial oid 47 2026-02-20T11:59:28.791 INFO:tasks.rados.rados.0.trial153.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2026-02-20T11:59:28.794 INFO:tasks.rados.rados.0.trial153.stdout:47: writing trial15342161-47 from 773122 to 1332694 tid 1 2026-02-20T11:59:28.799 INFO:tasks.rados.rados.0.trial153.stdout:47: writing trial15342161-47 from 1992054 to 2643735 tid 2 2026-02-20T11:59:28.799 INFO:tasks.rados.rados.0.trial153.stdout:48: write initial oid 48 2026-02-20T11:59:28.799 INFO:tasks.rados.rados.0.trial153.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2026-02-20T11:59:28.804 INFO:tasks.rados.rados.0.trial153.stdout:48: writing trial15342161-48 from 789929 to 1424750 tid 1 2026-02-20T11:59:28.810 INFO:tasks.rados.rados.0.trial153.stdout:48: writing trial15342161-48 from 2050536 to 2789929 tid 2 2026-02-20T11:59:28.810 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.810 INFO:tasks.rados.rados.0.trial153.stdout:42: finishing write tid 1 to trial15342161-42 2026-02-20T11:59:28.810 INFO:tasks.rados.rados.0.trial153.stdout:31: finishing write tid 1 to trial15342161-31 2026-02-20T11:59:28.810 INFO:tasks.rados.rados.0.trial153.stdout:43: finishing write tid 2 to trial15342161-43 2026-02-20T11:59:28.811 INFO:tasks.rados.rados.0.trial153.stdout:33: finishing write tid 2 to trial15342161-33 2026-02-20T11:59:28.812 INFO:tasks.rados.rados.0.trial153.stdout:46: finishing write tid 1 to trial15342161-46 2026-02-20T11:59:28.813 INFO:tasks.rados.rados.0.trial153.stdout:39: finishing write tid 2 to trial15342161-39 2026-02-20T11:59:28.816 INFO:tasks.rados.rados.0.trial153.stdout:39: finishing write tid 3 to trial15342161-39 2026-02-20T11:59:28.817 INFO:tasks.rados.rados.0.trial153.stdout:43: finishing write tid 3 to trial15342161-43 2026-02-20T11:59:28.819 INFO:tasks.rados.rados.0.trial153.stdout:39: finishing write tid 4 to trial15342161-39 2026-02-20T11:59:28.819 INFO:tasks.rados.rados.0.trial153.stdout:39: oid 39 updating version 0 to 9 2026-02-20T11:59:28.819 INFO:tasks.rados.rados.0.trial153.stdout:39: oid 39 version 9 is already newer than 7 2026-02-20T11:59:28.819 INFO:tasks.rados.rados.0.trial153.stdout:39: oid 39 version 9 is already newer than 9 2026-02-20T11:59:28.820 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 39 v 9 (ObjNum 38 snap 0 seq_num 38) dirty exists 2026-02-20T11:59:28.820 INFO:tasks.rados.rados.0.trial153.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2026-02-20T11:59:28.820 INFO:tasks.rados.rados.0.trial153.stdout:39: done (15 left) 2026-02-20T11:59:28.820 INFO:tasks.rados.rados.0.trial153.stdout:49: write initial oid 49 2026-02-20T11:59:28.820 INFO:tasks.rados.rados.0.trial153.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2026-02-20T11:59:28.823 INFO:tasks.rados.rados.0.trial153.stdout:49: writing trial15342161-49 from 406736 to 1116806 tid 1 2026-02-20T11:59:28.829 INFO:tasks.rados.rados.0.trial153.stdout:49: writing trial15342161-49 from 1709018 to 2364368 tid 2 2026-02-20T11:59:28.829 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.829 INFO:tasks.rados.rados.0.trial153.stdout:43: finishing write tid 4 to trial15342161-43 2026-02-20T11:59:28.829 INFO:tasks.rados.rados.0.trial153.stdout:43: oid 43 updating version 0 to 3 2026-02-20T11:59:28.829 INFO:tasks.rados.rados.0.trial153.stdout:43: oid 43 version 3 is already newer than 1 2026-02-20T11:59:28.830 INFO:tasks.rados.rados.0.trial153.stdout:43: oid 43 version 3 is already newer than 3 2026-02-20T11:59:28.830 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 43 v 3 (ObjNum 42 snap 0 seq_num 42) dirty exists 2026-02-20T11:59:28.830 INFO:tasks.rados.rados.0.trial153.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2026-02-20T11:59:28.830 INFO:tasks.rados.rados.0.trial153.stdout:43: done (15 left) 2026-02-20T11:59:28.830 INFO:tasks.rados.rados.0.trial153.stdout:50: write initial oid 50 2026-02-20T11:59:28.830 INFO:tasks.rados.rados.0.trial153.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2026-02-20T11:59:28.833 INFO:tasks.rados.rados.0.trial153.stdout:50: writing trial15342161-50 from 423543 to 1208862 tid 1 2026-02-20T11:59:28.839 INFO:tasks.rados.rados.0.trial153.stdout:50: writing trial15342161-50 from 1767500 to 2566508 tid 2 2026-02-20T11:59:28.839 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:42: finishing write tid 2 to trial15342161-42 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:42: finishing write tid 3 to trial15342161-42 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:33: finishing write tid 3 to trial15342161-33 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:48: finishing write tid 1 to trial15342161-48 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:42: finishing write tid 4 to trial15342161-42 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:42: oid 42 updating version 0 to 7 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:42: oid 42 updating version 7 to 9 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:42: oid 42 version 9 is already newer than 9 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:45: finishing write tid 1 to trial15342161-45 2026-02-20T11:59:28.840 INFO:tasks.rados.rados.0.trial153.stdout:42: done (15 left) 2026-02-20T11:59:28.841 INFO:tasks.rados.rados.0.trial153.stdout:51: write (excl) oid 2 current snap is 0 2026-02-20T11:59:28.841 INFO:tasks.rados.rados.0.trial153.stdout:33: finishing write tid 4 to trial15342161-33 2026-02-20T11:59:28.841 INFO:tasks.rados.rados.0.trial153.stdout:33: oid 33 updating version 0 to 4 2026-02-20T11:59:28.841 INFO:tasks.rados.rados.0.trial153.stdout:33: oid 33 updating version 4 to 6 2026-02-20T11:59:28.841 INFO:tasks.rados.rados.0.trial153.stdout:33: oid 33 version 6 is already newer than 6 2026-02-20T11:59:28.841 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2026-02-20T11:59:28.841 INFO:tasks.rados.rados.0.trial153.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2026-02-20T11:59:28.841 INFO:tasks.rados.rados.0.trial153.stdout:46: finishing write tid 2 to trial15342161-46 2026-02-20T11:59:28.841 INFO:tasks.rados.rados.0.trial153.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2026-02-20T11:59:28.842 INFO:tasks.rados.rados.0.trial153.stdout:51: writing trial15342161-2 from 440350 to 900918 tid 1 2026-02-20T11:59:28.845 INFO:tasks.rados.rados.0.trial153.stdout:51: writing trial15342161-2 from 1309629 to 1852295 tid 2 2026-02-20T11:59:28.850 INFO:tasks.rados.rados.0.trial153.stdout:51: writing trial15342161-2 from 2323973 to 2840350 tid 3 2026-02-20T11:59:28.850 INFO:tasks.rados.rados.0.trial153.stdout:33: done (15 left) 2026-02-20T11:59:28.850 INFO:tasks.rados.rados.0.trial153.stdout:52: write (excl) oid 14 current snap is 0 2026-02-20T11:59:28.850 INFO:tasks.rados.rados.0.trial153.stdout:31: finishing write tid 2 to trial15342161-31 2026-02-20T11:59:28.850 INFO:tasks.rados.rados.0.trial153.stdout:35: finishing write tid 3 to trial15342161-35 2026-02-20T11:59:28.850 INFO:tasks.rados.rados.0.trial153.stdout:46: finishing write tid 3 to trial15342161-46 2026-02-20T11:59:28.850 INFO:tasks.rados.rados.0.trial153.stdout:35: finishing write tid 4 to trial15342161-35 2026-02-20T11:59:28.850 INFO:tasks.rados.rados.0.trial153.stdout:35: oid 35 updating version 0 to 10 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:35: oid 35 updating version 10 to 12 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:35: oid 35 version 12 is already newer than 12 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:46: finishing write tid 4 to trial15342161-46 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:46: oid 46 updating version 0 to 6 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:46: oid 46 version 6 is already newer than 4 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:46: oid 46 version 6 is already newer than 6 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:49: finishing write tid 1 to trial15342161-49 2026-02-20T11:59:28.851 INFO:tasks.rados.rados.0.trial153.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2026-02-20T11:59:28.853 INFO:tasks.rados.rados.0.trial153.stdout:52: writing trial15342161-14 from 457157 to 992974 tid 1 2026-02-20T11:59:28.858 INFO:tasks.rados.rados.0.trial153.stdout:52: writing trial15342161-14 from 1768111 to 2170788 tid 2 2026-02-20T11:59:28.858 INFO:tasks.rados.rados.0.trial153.stdout:35: done (15 left) 2026-02-20T11:59:28.858 INFO:tasks.rados.rados.0.trial153.stdout:46: done (14 left) 2026-02-20T11:59:28.858 INFO:tasks.rados.rados.0.trial153.stdout:53: read oid 17 snap -1 2026-02-20T11:59:28.858 INFO:tasks.rados.rados.0.trial153.stdout:53: expect (ObjNum 16 snap 0 seq_num 16) 2026-02-20T11:59:28.859 INFO:tasks.rados.rados.0.trial153.stdout:54: read oid 35 snap -1 2026-02-20T11:59:28.859 INFO:tasks.rados.rados.0.trial153.stdout:54: expect (ObjNum 34 snap 0 seq_num 34) 2026-02-20T11:59:28.859 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.859 INFO:tasks.rados.rados.0.trial153.stdout:48: finishing write tid 2 to trial15342161-48 2026-02-20T11:59:28.859 INFO:tasks.rados.rados.0.trial153.stdout:21: finishing write tid 1 to trial15342161-21 2026-02-20T11:59:28.862 INFO:tasks.rados.rados.0.trial153.stdout:31: finishing write tid 3 to trial15342161-31 2026-02-20T11:59:28.865 INFO:tasks.rados.rados.0.trial153.stdout:31: finishing write tid 4 to trial15342161-31 2026-02-20T11:59:28.865 INFO:tasks.rados.rados.0.trial153.stdout:31: oid 31 updating version 0 to 9 2026-02-20T11:59:28.865 INFO:tasks.rados.rados.0.trial153.stdout:31: oid 31 version 9 is already newer than 9 2026-02-20T11:59:28.865 INFO:tasks.rados.rados.0.trial153.stdout:31: oid 31 version 9 is already newer than 7 2026-02-20T11:59:28.865 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 31 v 9 (ObjNum 30 snap 0 seq_num 30) dirty exists 2026-02-20T11:59:28.865 INFO:tasks.rados.rados.0.trial153.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2026-02-20T11:59:28.865 INFO:tasks.rados.rados.0.trial153.stdout:31: done (15 left) 2026-02-20T11:59:28.866 INFO:tasks.rados.rados.0.trial153.stdout:55: read oid 25 snap -1 2026-02-20T11:59:28.866 INFO:tasks.rados.rados.0.trial153.stdout:55: expect (ObjNum 24 snap 0 seq_num 24) 2026-02-20T11:59:28.866 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.866 INFO:tasks.rados.rados.0.trial153.stdout:48: finishing write tid 3 to trial15342161-48 2026-02-20T11:59:28.866 INFO:tasks.rados.rados.0.trial153.stdout:49: finishing write tid 2 to trial15342161-49 2026-02-20T11:59:28.868 INFO:tasks.rados.rados.0.trial153.stdout:48: finishing write tid 4 to trial15342161-48 2026-02-20T11:59:28.868 INFO:tasks.rados.rados.0.trial153.stdout:48: oid 48 updating version 0 to 6 2026-02-20T11:59:28.868 INFO:tasks.rados.rados.0.trial153.stdout:48: oid 48 version 6 is already newer than 4 2026-02-20T11:59:28.868 INFO:tasks.rados.rados.0.trial153.stdout:48: oid 48 version 6 is already newer than 6 2026-02-20T11:59:28.868 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 48 v 6 (ObjNum 47 snap 0 seq_num 47) dirty exists 2026-02-20T11:59:28.868 INFO:tasks.rados.rados.0.trial153.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2026-02-20T11:59:28.868 INFO:tasks.rados.rados.0.trial153.stdout:48: done (15 left) 2026-02-20T11:59:28.868 INFO:tasks.rados.rados.0.trial153.stdout:56: read oid 29 snap -1 2026-02-20T11:59:28.868 INFO:tasks.rados.rados.0.trial153.stdout:56: expect (ObjNum 28 snap 0 seq_num 28) 2026-02-20T11:59:28.868 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.869 INFO:tasks.rados.rados.0.trial153.stdout:51: finishing write tid 1 to trial15342161-2 2026-02-20T11:59:28.871 INFO:tasks.rados.rados.0.trial153.stdout:45: finishing write tid 2 to trial15342161-45 2026-02-20T11:59:28.875 INFO:tasks.rados.rados.0.trial153.stdout:49: finishing write tid 3 to trial15342161-49 2026-02-20T11:59:28.877 INFO:tasks.rados.rados.0.trial153.stdout:49: finishing write tid 4 to trial15342161-49 2026-02-20T11:59:28.878 INFO:tasks.rados.rados.0.trial153.stdout:49: oid 49 updating version 0 to 12 2026-02-20T11:59:28.878 INFO:tasks.rados.rados.0.trial153.stdout:49: oid 49 version 12 is already newer than 10 2026-02-20T11:59:28.878 INFO:tasks.rados.rados.0.trial153.stdout:49: oid 49 version 12 is already newer than 12 2026-02-20T11:59:28.878 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2026-02-20T11:59:28.878 INFO:tasks.rados.rados.0.trial153.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2026-02-20T11:59:28.878 INFO:tasks.rados.rados.0.trial153.stdout:49: done (15 left) 2026-02-20T11:59:28.878 INFO:tasks.rados.rados.0.trial153.stdout:57: read oid 28 snap -1 2026-02-20T11:59:28.878 INFO:tasks.rados.rados.0.trial153.stdout:57: expect (ObjNum 27 snap 0 seq_num 27) 2026-02-20T11:59:28.878 INFO:tasks.rados.rados.0.trial153.stdout: waiting on 16 2026-02-20T11:59:28.879 INFO:tasks.rados.rados.0.trial153.stdout:36: finishing write tid 1 to trial15342161-36 2026-02-20T11:59:28.880 INFO:tasks.rados.rados.0.trial153.stdout:53: expect (ObjNum 16 snap 0 seq_num 16) 2026-02-20T11:59:28.880 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:28.907 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1923050 2026-02-20T11:59:28.909 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:28.929 INFO:tasks.rados.rados.0.trial153.stdout:53: done (15 left) 2026-02-20T11:59:28.929 INFO:tasks.rados.rados.0.trial153.stdout:58: delete oid 6 current snap is 0 2026-02-20T11:59:28.929 INFO:tasks.rados.rados.0.trial153.stdout:51: finishing write tid 2 to trial15342161-2 2026-02-20T11:59:28.929 INFO:tasks.rados.rados.0.trial153.stdout:51: finishing write tid 3 to trial15342161-2 2026-02-20T11:59:28.929 INFO:tasks.rados.rados.0.trial153.stdout:21: finishing write tid 2 to trial15342161-21 2026-02-20T11:59:28.930 INFO:tasks.rados.rados.0.trial153.stdout:45: finishing write tid 3 to trial15342161-45 2026-02-20T11:59:28.930 INFO:tasks.rados.rados.0.trial153.stdout:45: finishing write tid 4 to trial15342161-45 2026-02-20T11:59:28.930 INFO:tasks.rados.rados.0.trial153.stdout:45: oid 45 updating version 0 to 4 2026-02-20T11:59:28.930 INFO:tasks.rados.rados.0.trial153.stdout:45: oid 45 updating version 4 to 6 2026-02-20T11:59:28.930 INFO:tasks.rados.rados.0.trial153.stdout:45: oid 45 version 6 is already newer than 6 2026-02-20T11:59:28.930 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2026-02-20T11:59:28.930 INFO:tasks.rados.rados.0.trial153.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2026-02-20T11:59:28.930 INFO:tasks.rados.rados.0.trial153.stdout:51: finishing write tid 4 to trial15342161-2 2026-02-20T11:59:28.930 INFO:tasks.rados.rados.0.trial153.stdout:56: expect (ObjNum 28 snap 0 seq_num 28) 2026-02-20T11:59:28.931 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 548649 2026-02-20T11:59:28.949 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 647168 2026-02-20T11:59:28.986 INFO:tasks.rados.rados.0.trial153.stdout:51: finishing write tid 5 to trial15342161-2 2026-02-20T11:59:28.986 INFO:tasks.rados.rados.0.trial153.stdout:51: oid 2 updating version 0 to 19 2026-02-20T11:59:28.986 INFO:tasks.rados.rados.0.trial153.stdout:51: oid 2 version 19 is already newer than 19 2026-02-20T11:59:28.986 INFO:tasks.rados.rados.0.trial153.stdout:51: oid 2 version 19 is already newer than 16 2026-02-20T11:59:28.986 INFO:tasks.rados.rados.0.trial153.stdout:51: oid 2 version 19 is already newer than 19 2026-02-20T11:59:28.986 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 2 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2026-02-20T11:59:28.986 INFO:tasks.rados.rados.0.trial153.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2026-02-20T11:59:28.987 INFO:tasks.rados.rados.0.trial153.stdout:21: finishing write tid 3 to trial15342161-21 2026-02-20T11:59:28.987 INFO:tasks.rados.rados.0.trial153.stdout:40: finishing write tid 1 to trial15342161-40 2026-02-20T11:59:28.987 INFO:tasks.rados.rados.0.trial153.stdout:21: finishing write tid 4 to trial15342161-21 2026-02-20T11:59:28.987 INFO:tasks.rados.rados.0.trial153.stdout:21: oid 21 updating version 0 to 9 2026-02-20T11:59:28.987 INFO:tasks.rados.rados.0.trial153.stdout:21: oid 21 version 9 is already newer than 7 2026-02-20T11:59:28.987 INFO:tasks.rados.rados.0.trial153.stdout:21: oid 21 version 9 is already newer than 9 2026-02-20T11:59:28.987 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2026-02-20T11:59:28.987 INFO:tasks.rados.rados.0.trial153.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2026-02-20T11:59:28.987 INFO:tasks.rados.rados.0.trial153.stdout:55: expect (ObjNum 24 snap 0 seq_num 24) 2026-02-20T11:59:29.003 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1233906 2026-02-20T11:59:29.016 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:29.036 INFO:tasks.rados.rados.0.trial153.stdout:36: finishing write tid 2 to trial15342161-36 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:50: finishing write tid 1 to trial15342161-50 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:40: finishing write tid 2 to trial15342161-40 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:36: finishing write tid 3 to trial15342161-36 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:36: finishing write tid 4 to trial15342161-36 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:36: oid 36 updating version 0 to 9 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:36: oid 36 version 9 is already newer than 9 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:36: oid 36 version 9 is already newer than 7 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:30: finishing write tid 1 to trial15342161-30 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:40: finishing write tid 3 to trial15342161-40 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:40: finishing write tid 4 to trial15342161-40 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:40: oid 40 updating version 0 to 15 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:40: oid 40 version 15 is already newer than 13 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:40: oid 40 version 15 is already newer than 15 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 40 v 15 (ObjNum 39 snap 0 seq_num 39) dirty exists 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:50: finishing write tid 2 to trial15342161-50 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:50: finishing write tid 3 to trial15342161-50 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:50: finishing write tid 4 to trial15342161-50 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:50: oid 50 updating version 0 to 13 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:50: oid 50 updating version 13 to 15 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:50: oid 50 version 15 is already newer than 15 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 50 v 15 (ObjNum 49 snap 0 seq_num 49) dirty exists 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:30: finishing write tid 2 to trial15342161-30 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:30: finishing write tid 3 to trial15342161-30 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:47: finishing write tid 1 to trial15342161-47 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:30: finishing write tid 4 to trial15342161-30 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:30: oid 30 updating version 0 to 12 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:30: oid 30 version 12 is already newer than 12 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:30: oid 30 version 12 is already newer than 10 2026-02-20T11:59:29.037 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2026-02-20T11:59:29.038 INFO:tasks.rados.rados.0.trial153.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2026-02-20T11:59:29.038 INFO:tasks.rados.rados.0.trial153.stdout:52: finishing write tid 1 to trial15342161-14 2026-02-20T11:59:29.038 INFO:tasks.rados.rados.0.trial153.stdout:47: finishing write tid 2 to trial15342161-47 2026-02-20T11:59:29.038 INFO:tasks.rados.rados.0.trial153.stdout:34: finishing write tid 1 to trial15342161-34 2026-02-20T11:59:29.038 INFO:tasks.rados.rados.0.trial153.stdout:54: expect (ObjNum 34 snap 0 seq_num 34) 2026-02-20T11:59:29.062 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2164892 2026-02-20T11:59:29.078 INFO:tasks.rados.rados.0.trial153.stdout:21: done (15 left) 2026-02-20T11:59:29.078 INFO:tasks.rados.rados.0.trial153.stdout:30: done (14 left) 2026-02-20T11:59:29.078 INFO:tasks.rados.rados.0.trial153.stdout:36: done (13 left) 2026-02-20T11:59:29.078 INFO:tasks.rados.rados.0.trial153.stdout:40: done (12 left) 2026-02-20T11:59:29.078 INFO:tasks.rados.rados.0.trial153.stdout:45: done (11 left) 2026-02-20T11:59:29.078 INFO:tasks.rados.rados.0.trial153.stdout:50: done (10 left) 2026-02-20T11:59:29.078 INFO:tasks.rados.rados.0.trial153.stdout:51: done (9 left) 2026-02-20T11:59:29.079 INFO:tasks.rados.rados.0.trial153.stdout:54: done (8 left) 2026-02-20T11:59:29.079 INFO:tasks.rados.rados.0.trial153.stdout:55: done (7 left) 2026-02-20T11:59:29.079 INFO:tasks.rados.rados.0.trial153.stdout:56: done (6 left) 2026-02-20T11:59:29.079 INFO:tasks.rados.rados.0.trial153.stdout:58: done (5 left) 2026-02-20T11:59:29.079 INFO:tasks.rados.rados.0.trial153.stdout:59: write oid 23 current snap is 0 2026-02-20T11:59:29.079 INFO:tasks.rados.rados.0.trial153.stdout:47: finishing write tid 3 to trial15342161-47 2026-02-20T11:59:29.079 INFO:tasks.rados.rados.0.trial153.stdout:59: seq_num 53 ranges {490771=686315,1885075=406346} 2026-02-20T11:59:29.081 INFO:tasks.rados.rados.0.trial153.stdout:59: writing trial15342161-23 from 490771 to 1177086 tid 1 2026-02-20T11:59:29.086 INFO:tasks.rados.rados.0.trial153.stdout:59: writing trial15342161-23 from 1885075 to 2291421 tid 2 2026-02-20T11:59:29.086 INFO:tasks.rados.rados.0.trial153.stdout:60: write oid 19 current snap is 0 2026-02-20T11:59:29.086 INFO:tasks.rados.rados.0.trial153.stdout:60: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2026-02-20T11:59:29.089 INFO:tasks.rados.rados.0.trial153.stdout:60: writing trial15342161-19 from 507578 to 985495 tid 1 2026-02-20T11:59:29.094 INFO:tasks.rados.rados.0.trial153.stdout:60: writing trial15342161-19 from 1543557 to 2093561 tid 2 2026-02-20T11:59:29.097 INFO:tasks.rados.rados.0.trial153.stdout:60: writing trial15342161-19 from 2833665 to 2907578 tid 3 2026-02-20T11:59:29.098 INFO:tasks.rados.rados.0.trial153.stdout:61: read oid 18 snap -1 2026-02-20T11:59:29.098 INFO:tasks.rados.rados.0.trial153.stdout:61: expect (ObjNum 17 snap 0 seq_num 17) 2026-02-20T11:59:29.098 INFO:tasks.rados.rados.0.trial153.stdout:62: snap_create 2026-02-20T11:59:29.098 INFO:tasks.rados.rados.0.trial153.stdout:47: finishing write tid 4 to trial15342161-47 2026-02-20T11:59:29.098 INFO:tasks.rados.rados.0.trial153.stdout:47: oid 47 updating version 0 to 18 2026-02-20T11:59:29.098 INFO:tasks.rados.rados.0.trial153.stdout:47: oid 47 version 18 is already newer than 18 2026-02-20T11:59:29.098 INFO:tasks.rados.rados.0.trial153.stdout:47: oid 47 version 18 is already newer than 16 2026-02-20T11:59:29.098 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 47 v 18 (ObjNum 46 snap 0 seq_num 46) dirty exists 2026-02-20T11:59:29.098 INFO:tasks.rados.rados.0.trial153.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2026-02-20T11:59:29.098 INFO:tasks.rados.rados.0.trial153.stdout:57: expect (ObjNum 27 snap 0 seq_num 27) 2026-02-20T11:59:29.114 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:29.156 INFO:tasks.rados.rados.0.trial153.stdout:52: finishing write tid 2 to trial15342161-14 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:52: finishing write tid 3 to trial15342161-14 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:52: finishing write tid 4 to trial15342161-14 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:52: oid 14 updating version 0 to 18 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:52: oid 14 version 18 is already newer than 18 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:52: oid 14 version 18 is already newer than 16 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 18 (ObjNum 51 snap 0 seq_num 51) dirty exists 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:52: left oid 14 (ObjNum 51 snap 0 seq_num 51) 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:34: finishing write tid 2 to trial15342161-34 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:34: finishing write tid 3 to trial15342161-34 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:34: finishing write tid 4 to trial15342161-34 2026-02-20T11:59:29.157 INFO:tasks.rados.rados.0.trial153.stdout:34: oid 34 updating version 0 to 15 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:34: oid 34 version 15 is already newer than 15 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:34: oid 34 version 15 is already newer than 13 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 15 (ObjNum 33 snap 0 seq_num 33) dirty exists 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:44: finishing write tid 1 to trial15342161-44 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:44: finishing write tid 2 to trial15342161-44 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:44: finishing write tid 3 to trial15342161-44 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:44: finishing write tid 4 to trial15342161-44 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:44: oid 44 updating version 0 to 16 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:44: oid 44 updating version 16 to 18 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:44: oid 44 version 18 is already newer than 18 2026-02-20T11:59:29.158 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 44 v 18 (ObjNum 43 snap 0 seq_num 43) dirty exists 2026-02-20T11:59:29.159 INFO:tasks.rados.rados.0.trial153.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2026-02-20T11:59:29.159 INFO:tasks.rados.rados.0.trial153.stdout:61: expect (ObjNum 17 snap 0 seq_num 17) 2026-02-20T11:59:29.199 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1180408 2026-02-20T11:59:29.208 INFO:tasks.rados.rados.0.trial153.stdout:59: finishing write tid 1 to trial15342161-23 2026-02-20T11:59:29.208 INFO:tasks.rados.rados.0.trial153.stdout:60: finishing write tid 1 to trial15342161-19 2026-02-20T11:59:29.208 INFO:tasks.rados.rados.0.trial153.stdout:60: finishing write tid 2 to trial15342161-19 2026-02-20T11:59:29.208 INFO:tasks.rados.rados.0.trial153.stdout:60: finishing write tid 3 to trial15342161-19 2026-02-20T11:59:29.208 INFO:tasks.rados.rados.0.trial153.stdout:60: finishing write tid 4 to trial15342161-19 2026-02-20T11:59:29.208 INFO:tasks.rados.rados.0.trial153.stdout:60: finishing write tid 5 to trial15342161-19 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:60: oid 19 updating version 0 to 23 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:60: oid 19 version 23 is already newer than 23 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:60: oid 19 version 23 is already newer than 23 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:60: oid 19 version 23 is already newer than 23 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 23 (ObjNum 54 snap 0 seq_num 54) dirty exists 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:60: left oid 19 (ObjNum 54 snap 0 seq_num 54) 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:59: finishing write tid 2 to trial15342161-23 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:59: finishing write tid 3 to trial15342161-23 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:59: finishing write tid 4 to trial15342161-23 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:59: oid 23 updating version 0 to 21 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:59: oid 23 version 21 is already newer than 21 2026-02-20T11:59:29.209 INFO:tasks.rados.rados.0.trial153.stdout:59: oid 23 version 21 is already newer than 21 2026-02-20T11:59:29.210 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 23 v 21 (ObjNum 53 snap 0 seq_num 53) dirty exists 2026-02-20T11:59:29.210 INFO:tasks.rados.rados.0.trial153.stdout:59: left oid 23 (ObjNum 53 snap 0 seq_num 53) 2026-02-20T11:59:29.501 INFO:tasks.rados.rados.0.trial153.stdout:34: done (8 left) 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:44: done (7 left) 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:47: done (6 left) 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:52: done (5 left) 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:57: done (4 left) 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:59: done (3 left) 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:60: done (2 left) 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:61: done (1 left) 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:62: done (0 left) 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:63: read oid 49 snap -1 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:63: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-20T11:59:29.502 INFO:tasks.rados.rados.0.trial153.stdout:64: delete oid 24 current snap is 1 2026-02-20T11:59:29.526 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:59:29.527 INFO:tasks.rados.rados.0.trial153.stdout:63: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-20T11:59:29.599 INFO:tasks.rados.rados.0.trial153.stdout:63: done (1 left) 2026-02-20T11:59:29.599 INFO:tasks.rados.rados.0.trial153.stdout:64: done (0 left) 2026-02-20T11:59:29.599 INFO:tasks.rados.rados.0.trial153.stdout:65: read oid 39 snap 0 2026-02-20T11:59:29.599 INFO:tasks.rados.rados.0.trial153.stdout:65: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-20T11:59:29.599 INFO:tasks.rados.rados.0.trial153.stdout:66: read oid 16 snap -1 2026-02-20T11:59:29.600 INFO:tasks.rados.rados.0.trial153.stdout:66: expect (ObjNum 15 snap 0 seq_num 15) 2026-02-20T11:59:29.600 INFO:tasks.rados.rados.0.trial153.stdout:67: write oid 45 current snap is 1 2026-02-20T11:59:29.600 INFO:tasks.rados.rados.0.trial153.stdout:67: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2026-02-20T11:59:29.602 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:29.596+0000 7fd8819cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:29.603 INFO:tasks.rados.rados.0.trial153.stdout:67: writing trial15342161-45 from 541192 to 1169607 tid 1 2026-02-20T11:59:29.605 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:29.600+0000 7fd8819cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:29.607 INFO:tasks.rados.rados.0.trial153.stdout:67: writing trial15342161-45 from 1660521 to 2214194 tid 2 2026-02-20T11:59:29.610 INFO:tasks.rados.rados.0.trial153.stdout:67: writing trial15342161-45 from 2888511 to 2941192 tid 3 2026-02-20T11:59:29.610 INFO:tasks.rados.rados.0.trial153.stdout:68: read oid 48 snap -1 2026-02-20T11:59:29.610 INFO:tasks.rados.rados.0.trial153.stdout:68: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-20T11:59:29.610 INFO:tasks.rados.rados.0.trial153.stdout:69: snap_create 2026-02-20T11:59:29.615 INFO:tasks.rados.rados.0.trial153.stdout:66: expect (ObjNum 15 snap 0 seq_num 15) 2026-02-20T11:59:29.616 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:29.655 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:29.670 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:29.671 INFO:tasks.rados.rados.0.trial153.stdout:65: expect (ObjNum 38 snap 0 seq_num 38) 2026-02-20T11:59:29.671 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:29.712 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:29.734 INFO:tasks.rados.rados.0.trial153.stdout:68: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-20T11:59:29.735 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:29.772 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 995082 2026-02-20T11:59:29.786 INFO:tasks.rados.rados.0.trial153.stdout:67: finishing write tid 1 to trial15342161-45 2026-02-20T11:59:29.786 INFO:tasks.rados.rados.0.trial153.stdout:67: finishing write tid 2 to trial15342161-45 2026-02-20T11:59:29.786 INFO:tasks.rados.rados.0.trial153.stdout:67: finishing write tid 3 to trial15342161-45 2026-02-20T11:59:29.786 INFO:tasks.rados.rados.0.trial153.stdout:67: finishing write tid 4 to trial15342161-45 2026-02-20T11:59:29.786 INFO:tasks.rados.rados.0.trial153.stdout:67: finishing write tid 5 to trial15342161-45 2026-02-20T11:59:29.787 INFO:tasks.rados.rados.0.trial153.stdout:67: oid 45 updating version 0 to 11 2026-02-20T11:59:29.787 INFO:tasks.rados.rados.0.trial153.stdout:67: oid 45 version 11 is already newer than 11 2026-02-20T11:59:29.787 INFO:tasks.rados.rados.0.trial153.stdout:67: oid 45 version 11 is already newer than 11 2026-02-20T11:59:29.787 INFO:tasks.rados.rados.0.trial153.stdout:67: oid 45 version 11 is already newer than 11 2026-02-20T11:59:29.787 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 11 (ObjNum 56 snap 1 seq_num 56) dirty exists 2026-02-20T11:59:29.787 INFO:tasks.rados.rados.0.trial153.stdout:67: left oid 45 (ObjNum 56 snap 1 seq_num 56) 2026-02-20T11:59:30.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~17s 2026-02-20T11:59:30.504 INFO:tasks.rados.rados.0.trial153.stdout:65: done (4 left) 2026-02-20T11:59:30.504 INFO:tasks.rados.rados.0.trial153.stdout:66: done (3 left) 2026-02-20T11:59:30.504 INFO:tasks.rados.rados.0.trial153.stdout:67: done (2 left) 2026-02-20T11:59:30.504 INFO:tasks.rados.rados.0.trial153.stdout:68: done (1 left) 2026-02-20T11:59:30.504 INFO:tasks.rados.rados.0.trial153.stdout:69: done (0 left) 2026-02-20T11:59:30.504 INFO:tasks.rados.rados.0.trial153.stdout:70: delete oid 38 current snap is 2 2026-02-20T11:59:30.535 INFO:tasks.rados.rados.0.trial153.stdout:70: done (0 left) 2026-02-20T11:59:30.535 INFO:tasks.rados.rados.0.trial153.stdout:71: snap_remove snap 1 2026-02-20T11:59:31.510 INFO:tasks.rados.rados.0.trial153.stdout:71: done (0 left) 2026-02-20T11:59:31.510 INFO:tasks.rados.rados.0.trial153.stdout:72: read oid 15 snap -1 2026-02-20T11:59:31.510 INFO:tasks.rados.rados.0.trial153.stdout:72: expect (ObjNum 14 snap 0 seq_num 14) 2026-02-20T11:59:31.510 INFO:tasks.rados.rados.0.trial153.stdout:73: write oid 42 current snap is 2 2026-02-20T11:59:31.510 INFO:tasks.rados.rados.0.trial153.stdout:73: seq_num 58 ranges {574806=778913,2061132=557342} 2026-02-20T11:59:31.515 INFO:tasks.rados.rados.0.trial153.stdout:73: writing trial15342161-42 from 574806 to 1353719 tid 1 2026-02-20T11:59:31.520 INFO:tasks.rados.rados.0.trial153.stdout:73: writing trial15342161-42 from 2061132 to 2618474 tid 2 2026-02-20T11:59:31.520 INFO:tasks.rados.rados.0.trial153.stdout:74: write (excl) oid 29 current snap is 2 2026-02-20T11:59:31.520 INFO:tasks.rados.rados.0.trial153.stdout:74: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2026-02-20T11:59:31.523 INFO:tasks.rados.rados.0.trial153.stdout:74: writing trial15342161-29 from 591613 to 1045775 tid 1 2026-02-20T11:59:31.528 INFO:tasks.rados.rados.0.trial153.stdout:74: writing trial15342161-29 from 1719614 to 2136967 tid 2 2026-02-20T11:59:31.531 INFO:tasks.rados.rados.0.trial153.stdout:74: writing trial15342161-29 from 2570780 to 2991613 tid 3 2026-02-20T11:59:31.531 INFO:tasks.rados.rados.0.trial153.stdout:75: write (excl) oid 10 current snap is 2 2026-02-20T11:59:31.532 INFO:tasks.rados.rados.0.trial153.stdout:75: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2026-02-20T11:59:31.536 INFO:tasks.rados.rados.0.trial153.stdout:75: writing trial15342161-10 from 608420 to 1137831 tid 1 2026-02-20T11:59:31.541 INFO:tasks.rados.rados.0.trial153.stdout:75: writing trial15342161-10 from 1778096 to 2339107 tid 2 2026-02-20T11:59:31.543 INFO:tasks.rados.rados.0.trial153.stdout:75: writing trial15342161-10 from 2881850 to 3008420 tid 3 2026-02-20T11:59:31.543 INFO:tasks.rados.rados.0.trial153.stdout:76: snap_remove snap 0 2026-02-20T11:59:31.672 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:59:31.748 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:31.742+0000 7f9d6570b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:31.751 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:31.746+0000 7f9d6570b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:31.777 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:31.794 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:32.513 INFO:tasks.rados.rados.0.trial153.stdout:76: done (4 left) 2026-02-20T11:59:32.513 INFO:tasks.rados.rados.0.trial153.stdout:77: write oid 40 current snap is 2 2026-02-20T11:59:32.513 INFO:tasks.rados.rados.0.trial153.stdout:77: seq_num 61 ranges {625227=721013,1952931=704669} 2026-02-20T11:59:32.518 INFO:tasks.rados.rados.0.trial153.stdout:77: writing trial15342161-40 from 625227 to 1346240 tid 1 2026-02-20T11:59:32.523 INFO:tasks.rados.rados.0.trial153.stdout:77: writing trial15342161-40 from 1952931 to 2657600 tid 2 2026-02-20T11:59:32.523 INFO:tasks.rados.rados.0.trial153.stdout:78: snap_create 2026-02-20T11:59:32.523 INFO:tasks.rados.rados.0.trial153.stdout:72: expect (ObjNum 14 snap 0 seq_num 14) 2026-02-20T11:59:32.561 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:32.581 INFO:tasks.rados.rados.0.trial153.stdout:75: finishing write tid 1 to trial15342161-10 2026-02-20T11:59:32.582 INFO:tasks.rados.rados.0.trial153.stdout:73: finishing write tid 1 to trial15342161-42 2026-02-20T11:59:32.582 INFO:tasks.rados.rados.0.trial153.stdout:75: finishing write tid 2 to trial15342161-10 2026-02-20T11:59:32.582 INFO:tasks.rados.rados.0.trial153.stdout:73: finishing write tid 2 to trial15342161-42 2026-02-20T11:59:32.582 INFO:tasks.rados.rados.0.trial153.stdout:75: finishing write tid 3 to trial15342161-10 2026-02-20T11:59:32.582 INFO:tasks.rados.rados.0.trial153.stdout:75: finishing write tid 4 to trial15342161-10 2026-02-20T11:59:32.582 INFO:tasks.rados.rados.0.trial153.stdout:73: finishing write tid 3 to trial15342161-42 2026-02-20T11:59:32.582 INFO:tasks.rados.rados.0.trial153.stdout:75: finishing write tid 5 to trial15342161-10 2026-02-20T11:59:32.582 INFO:tasks.rados.rados.0.trial153.stdout:75: oid 10 updating version 0 to 20 2026-02-20T11:59:32.582 INFO:tasks.rados.rados.0.trial153.stdout:75: oid 10 updating version 20 to 23 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:75: oid 10 version 23 is already newer than 23 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:75: oid 10 version 23 is already newer than 23 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 10 v 23 (ObjNum 60 snap 2 seq_num 60) dirty exists 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:75: left oid 10 (ObjNum 60 snap 2 seq_num 60) 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:73: finishing write tid 4 to trial15342161-42 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:73: oid 42 updating version 0 to 13 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:73: oid 42 version 13 is already newer than 13 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:73: oid 42 version 13 is already newer than 13 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 13 (ObjNum 58 snap 2 seq_num 58) dirty exists 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:73: left oid 42 (ObjNum 58 snap 2 seq_num 58) 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:74: finishing write tid 1 to trial15342161-29 2026-02-20T11:59:32.583 INFO:tasks.rados.rados.0.trial153.stdout:74: finishing write tid 2 to trial15342161-29 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:74: finishing write tid 3 to trial15342161-29 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:74: finishing write tid 4 to trial15342161-29 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:74: finishing write tid 5 to trial15342161-29 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:74: oid 29 updating version 0 to 24 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:74: oid 29 version 24 is already newer than 21 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:74: oid 29 version 24 is already newer than 24 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:74: oid 29 version 24 is already newer than 24 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 29 v 24 (ObjNum 59 snap 2 seq_num 59) dirty exists 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:74: left oid 29 (ObjNum 59 snap 2 seq_num 59) 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:77: finishing write tid 1 to trial15342161-40 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:77: finishing write tid 2 to trial15342161-40 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:77: finishing write tid 3 to trial15342161-40 2026-02-20T11:59:32.584 INFO:tasks.rados.rados.0.trial153.stdout:77: finishing write tid 4 to trial15342161-40 2026-02-20T11:59:32.585 INFO:tasks.rados.rados.0.trial153.stdout:77: oid 40 updating version 0 to 24 2026-02-20T11:59:32.585 INFO:tasks.rados.rados.0.trial153.stdout:77: oid 40 version 24 is already newer than 24 2026-02-20T11:59:32.585 INFO:tasks.rados.rados.0.trial153.stdout:77: oid 40 version 24 is already newer than 24 2026-02-20T11:59:32.585 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 40 v 24 (ObjNum 61 snap 2 seq_num 61) dirty exists 2026-02-20T11:59:32.585 INFO:tasks.rados.rados.0.trial153.stdout:77: left oid 40 (ObjNum 61 snap 2 seq_num 61) 2026-02-20T11:59:32.613 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:32.683 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:32.686 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:33.519 INFO:tasks.rados.rados.0.trial153.stdout:72: done (5 left) 2026-02-20T11:59:33.519 INFO:tasks.rados.rados.0.trial153.stdout:73: done (4 left) 2026-02-20T11:59:33.519 INFO:tasks.rados.rados.0.trial153.stdout:74: done (3 left) 2026-02-20T11:59:33.519 INFO:tasks.rados.rados.0.trial153.stdout:75: done (2 left) 2026-02-20T11:59:33.519 INFO:tasks.rados.rados.0.trial153.stdout:77: done (1 left) 2026-02-20T11:59:33.519 INFO:tasks.rados.rados.0.trial153.stdout:78: done (0 left) 2026-02-20T11:59:33.519 INFO:tasks.rados.rados.0.trial153.stdout:79: write oid 6 current snap is 3 2026-02-20T11:59:33.520 INFO:tasks.rados.rados.0.trial153.stdout:79: seq_num 62 ranges {642034=796262,1895060=564680} 2026-02-20T11:59:33.523 INFO:tasks.rados.rados.0.trial153.stdout:79: writing trial15342161-6 from 642034 to 1438296 tid 1 2026-02-20T11:59:33.528 INFO:tasks.rados.rados.0.trial153.stdout:79: writing trial15342161-6 from 1895060 to 2459740 tid 2 2026-02-20T11:59:33.528 INFO:tasks.rados.rados.0.trial153.stdout:80: write oid 7 current snap is 3 2026-02-20T11:59:33.528 INFO:tasks.rados.rados.0.trial153.stdout:80: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2026-02-20T11:59:33.532 INFO:tasks.rados.rados.0.trial153.stdout:80: writing trial15342161-7 from 658841 to 1130352 tid 1 2026-02-20T11:59:33.536 INFO:tasks.rados.rados.0.trial153.stdout:80: writing trial15342161-7 from 1553542 to 2261880 tid 2 2026-02-20T11:59:33.539 INFO:tasks.rados.rados.0.trial153.stdout:80: writing trial15342161-7 from 2964119 to 3058841 tid 3 2026-02-20T11:59:33.539 INFO:tasks.rados.rados.0.trial153.stdout:81: write oid 22 current snap is 3 2026-02-20T11:59:33.539 INFO:tasks.rados.rados.0.trial153.stdout:81: seq_num 64 ranges {675648=546760,2012024=568349} 2026-02-20T11:59:33.543 INFO:tasks.rados.rados.0.trial153.stdout:81: writing trial15342161-22 from 675648 to 1222408 tid 1 2026-02-20T11:59:33.549 INFO:tasks.rados.rados.0.trial153.stdout:81: writing trial15342161-22 from 2012024 to 2580373 tid 2 2026-02-20T11:59:33.549 INFO:tasks.rados.rados.0.trial153.stdout:82: delete oid 14 current snap is 3 2026-02-20T11:59:33.550 INFO:tasks.rados.rados.0.trial153.stdout:79: finishing write tid 1 to trial15342161-6 2026-02-20T11:59:33.562 INFO:tasks.rados.rados.0.trial153.stdout:80: finishing write tid 1 to trial15342161-7 2026-02-20T11:59:33.565 INFO:tasks.rados.rados.0.trial153.stdout:79: finishing write tid 2 to trial15342161-6 2026-02-20T11:59:33.568 INFO:tasks.rados.rados.0.trial153.stdout:80: finishing write tid 2 to trial15342161-7 2026-02-20T11:59:33.572 INFO:tasks.rados.rados.0.trial153.stdout:80: finishing write tid 3 to trial15342161-7 2026-02-20T11:59:33.572 INFO:tasks.rados.rados.0.trial153.stdout:79: finishing write tid 3 to trial15342161-6 2026-02-20T11:59:33.576 INFO:tasks.rados.rados.0.trial153.stdout:80: finishing write tid 4 to trial15342161-7 2026-02-20T11:59:33.576 INFO:tasks.rados.rados.0.trial153.stdout:79: finishing write tid 4 to trial15342161-6 2026-02-20T11:59:33.577 INFO:tasks.rados.rados.0.trial153.stdout:79: oid 6 updating version 0 to 26 2026-02-20T11:59:33.577 INFO:tasks.rados.rados.0.trial153.stdout:79: oid 6 version 26 is already newer than 26 2026-02-20T11:59:33.577 INFO:tasks.rados.rados.0.trial153.stdout:79: oid 6 version 26 is already newer than 26 2026-02-20T11:59:33.577 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 6 v 26 (ObjNum 62 snap 3 seq_num 62) dirty exists 2026-02-20T11:59:33.577 INFO:tasks.rados.rados.0.trial153.stdout:79: left oid 6 (ObjNum 62 snap 3 seq_num 62) 2026-02-20T11:59:33.578 INFO:tasks.rados.rados.0.trial153.stdout:80: finishing write tid 5 to trial15342161-7 2026-02-20T11:59:33.578 INFO:tasks.rados.rados.0.trial153.stdout:80: oid 7 updating version 0 to 14 2026-02-20T11:59:33.578 INFO:tasks.rados.rados.0.trial153.stdout:80: oid 7 version 14 is already newer than 14 2026-02-20T11:59:33.578 INFO:tasks.rados.rados.0.trial153.stdout:80: oid 7 version 14 is already newer than 14 2026-02-20T11:59:33.578 INFO:tasks.rados.rados.0.trial153.stdout:80: oid 7 version 14 is already newer than 14 2026-02-20T11:59:33.578 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 7 v 14 (ObjNum 63 snap 3 seq_num 63) dirty exists 2026-02-20T11:59:33.578 INFO:tasks.rados.rados.0.trial153.stdout:80: left oid 7 (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:59:33.599 INFO:tasks.rados.rados.0.trial153.stdout:81: finishing write tid 1 to trial15342161-22 2026-02-20T11:59:33.609 INFO:tasks.rados.rados.0.trial153.stdout:81: finishing write tid 2 to trial15342161-22 2026-02-20T11:59:33.619 INFO:tasks.rados.rados.0.trial153.stdout:81: finishing write tid 3 to trial15342161-22 2026-02-20T11:59:33.621 INFO:tasks.rados.rados.0.trial153.stdout:81: finishing write tid 4 to trial15342161-22 2026-02-20T11:59:33.621 INFO:tasks.rados.rados.0.trial153.stdout:81: oid 22 updating version 0 to 28 2026-02-20T11:59:33.621 INFO:tasks.rados.rados.0.trial153.stdout:81: oid 22 version 28 is already newer than 28 2026-02-20T11:59:33.621 INFO:tasks.rados.rados.0.trial153.stdout:81: oid 22 version 28 is already newer than 28 2026-02-20T11:59:33.621 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 22 v 28 (ObjNum 64 snap 3 seq_num 64) dirty exists 2026-02-20T11:59:33.621 INFO:tasks.rados.rados.0.trial153.stdout:81: left oid 22 (ObjNum 64 snap 3 seq_num 64) 2026-02-20T11:59:33.629 INFO:tasks.rados.rados.0.trial153.stdout:79: done (3 left) 2026-02-20T11:59:33.630 INFO:tasks.rados.rados.0.trial153.stdout:80: done (2 left) 2026-02-20T11:59:33.630 INFO:tasks.rados.rados.0.trial153.stdout:81: done (1 left) 2026-02-20T11:59:33.630 INFO:tasks.rados.rados.0.trial153.stdout:82: done (0 left) 2026-02-20T11:59:33.630 INFO:tasks.rados.rados.0.trial153.stdout:83: write (excl) oid 44 current snap is 3 2026-02-20T11:59:33.630 INFO:tasks.rados.rados.0.trial153.stdout:83: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2026-02-20T11:59:33.635 INFO:tasks.rados.rados.0.trial153.stdout:83: writing trial15342161-44 from 709262 to 1406520 tid 1 2026-02-20T11:59:33.639 INFO:tasks.rados.rados.0.trial153.stdout:83: writing trial15342161-44 from 2012635 to 2584653 tid 2 2026-02-20T11:59:33.641 INFO:tasks.rados.rados.0.trial153.stdout:83: writing trial15342161-44 from 3046388 to 3109262 tid 3 2026-02-20T11:59:33.641 INFO:tasks.rados.rados.0.trial153.stdout:84: write oid 22 current snap is 3 2026-02-20T11:59:33.641 INFO:tasks.rados.rados.0.trial153.stdout:84: seq_num 67 ranges {726069=772507,2071117=715676} 2026-02-20T11:59:33.646 INFO:tasks.rados.rados.0.trial153.stdout:84: writing trial15342161-22 from 726069 to 1498576 tid 1 2026-02-20T11:59:33.652 INFO:tasks.rados.rados.0.trial153.stdout:84: writing trial15342161-22 from 2071117 to 2786793 tid 2 2026-02-20T11:59:33.652 INFO:tasks.rados.rados.0.trial153.stdout:85: read oid 23 snap -1 2026-02-20T11:59:33.652 INFO:tasks.rados.rados.0.trial153.stdout:85: expect (ObjNum 53 snap 0 seq_num 53) 2026-02-20T11:59:33.652 INFO:tasks.rados.rados.0.trial153.stdout:86: read oid 29 snap -1 2026-02-20T11:59:33.652 INFO:tasks.rados.rados.0.trial153.stdout:86: expect (ObjNum 59 snap 2 seq_num 59) 2026-02-20T11:59:33.652 INFO:tasks.rados.rados.0.trial153.stdout:87: write oid 42 current snap is 3 2026-02-20T11:59:33.652 INFO:tasks.rados.rados.0.trial153.stdout:87: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2026-02-20T11:59:33.656 INFO:tasks.rados.rados.0.trial153.stdout:87: writing trial15342161-42 from 742876 to 1190632 tid 1 2026-02-20T11:59:33.660 INFO:tasks.rados.rados.0.trial153.stdout:87: writing trial15342161-42 from 1729599 to 2305286 tid 2 2026-02-20T11:59:33.663 INFO:tasks.rados.rados.0.trial153.stdout:87: writing trial15342161-42 from 3101234 to 3142876 tid 3 2026-02-20T11:59:33.663 INFO:tasks.rados.rados.0.trial153.stdout:88: snap_remove snap 2 2026-02-20T11:59:33.795 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:59:33.871 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:33.864+0000 7f9d0dfdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:33.873 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:33.868+0000 7f9d0dfdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:33.899 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:33.916 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:34.519 INFO:tasks.rados.rados.0.trial153.stdout:88: done (5 left) 2026-02-20T11:59:34.519 INFO:tasks.rados.rados.0.trial153.stdout:89: write oid 4 current snap is 3 2026-02-20T11:59:34.519 INFO:tasks.rados.rados.0.trial153.stdout:83: finishing write tid 1 to trial15342161-44 2026-02-20T11:59:34.519 INFO:tasks.rados.rados.0.trial153.stdout:89: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2026-02-20T11:59:34.525 INFO:tasks.rados.rados.0.trial153.stdout:89: writing trial15342161-4 from 759683 to 1399041 tid 1 2026-02-20T11:59:34.530 INFO:tasks.rados.rados.0.trial153.stdout:89: writing trial15342161-4 from 1904434 to 2623779 tid 2 2026-02-20T11:59:34.532 INFO:tasks.rados.rados.0.trial153.stdout:89: writing trial15342161-4 from 3128657 to 3159683 tid 3 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:90: read oid 36 snap -1 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:90: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:91: read oid 33 snap -1 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:91: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:92: read oid 41 snap -1 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:92: expect (ObjNum 40 snap 0 seq_num 40) 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:93: read oid 31 snap -1 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:93: expect (ObjNum 30 snap 0 seq_num 30) 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:94: write oid 3 current snap is 3 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:84: finishing write tid 1 to trial15342161-22 2026-02-20T11:59:34.533 INFO:tasks.rados.rados.0.trial153.stdout:86: expect (ObjNum 59 snap 2 seq_num 59) 2026-02-20T11:59:34.581 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1773803 2026-02-20T11:59:34.598 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1288066 2026-02-20T11:59:34.622 INFO:tasks.rados.rados.0.trial153.stdout:84: finishing write tid 2 to trial15342161-22 2026-02-20T11:59:34.622 INFO:tasks.rados.rados.0.trial153.stdout:94: seq_num 70 ranges {776490=714607,2246563=579356} 2026-02-20T11:59:34.628 INFO:tasks.rados.rados.0.trial153.stdout:94: writing trial15342161-3 from 776490 to 1491097 tid 1 2026-02-20T11:59:34.633 INFO:tasks.rados.rados.0.trial153.stdout:94: writing trial15342161-3 from 2246563 to 2825919 tid 2 2026-02-20T11:59:34.634 INFO:tasks.rados.rados.0.trial153.stdout:86: done (10 left) 2026-02-20T11:59:34.634 INFO:tasks.rados.rados.0.trial153.stdout:95: delete oid 39 current snap is 3 2026-02-20T11:59:34.634 INFO:tasks.rados.rados.0.trial153.stdout:83: finishing write tid 2 to trial15342161-44 2026-02-20T11:59:34.634 INFO:tasks.rados.rados.0.trial153.stdout:87: finishing write tid 1 to trial15342161-42 2026-02-20T11:59:34.634 INFO:tasks.rados.rados.0.trial153.stdout:84: finishing write tid 3 to trial15342161-22 2026-02-20T11:59:34.634 INFO:tasks.rados.rados.0.trial153.stdout:84: finishing write tid 4 to trial15342161-22 2026-02-20T11:59:34.634 INFO:tasks.rados.rados.0.trial153.stdout:84: oid 22 updating version 0 to 31 2026-02-20T11:59:34.634 INFO:tasks.rados.rados.0.trial153.stdout:84: oid 22 version 31 is already newer than 31 2026-02-20T11:59:34.634 INFO:tasks.rados.rados.0.trial153.stdout:84: oid 22 version 31 is already newer than 31 2026-02-20T11:59:34.634 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 22 v 31 (ObjNum 67 snap 3 seq_num 67) dirty exists 2026-02-20T11:59:34.635 INFO:tasks.rados.rados.0.trial153.stdout:84: left oid 22 (ObjNum 67 snap 3 seq_num 67) 2026-02-20T11:59:34.635 INFO:tasks.rados.rados.0.trial153.stdout:83: finishing write tid 3 to trial15342161-44 2026-02-20T11:59:34.635 INFO:tasks.rados.rados.0.trial153.stdout:85: expect (ObjNum 53 snap 0 seq_num 53) 2026-02-20T11:59:34.675 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2114051 2026-02-20T11:59:34.686 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:34.721 INFO:tasks.rados.rados.0.trial153.stdout:87: finishing write tid 2 to trial15342161-42 2026-02-20T11:59:34.721 INFO:tasks.rados.rados.0.trial153.stdout:83: finishing write tid 4 to trial15342161-44 2026-02-20T11:59:34.721 INFO:tasks.rados.rados.0.trial153.stdout:84: done (10 left) 2026-02-20T11:59:34.721 INFO:tasks.rados.rados.0.trial153.stdout:85: done (9 left) 2026-02-20T11:59:34.721 INFO:tasks.rados.rados.0.trial153.stdout:95: done (8 left) 2026-02-20T11:59:34.721 INFO:tasks.rados.rados.0.trial153.stdout:96: read oid 48 snap -1 2026-02-20T11:59:34.721 INFO:tasks.rados.rados.0.trial153.stdout:96: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-20T11:59:34.721 INFO:tasks.rados.rados.0.trial153.stdout:97: write oid 23 current snap is 3 2026-02-20T11:59:34.721 INFO:tasks.rados.rados.0.trial153.stdout:83: finishing write tid 5 to trial15342161-44 2026-02-20T11:59:34.721 INFO:tasks.rados.rados.0.trial153.stdout:83: oid 44 updating version 0 to 28 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:83: oid 44 version 28 is already newer than 28 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:83: oid 44 version 28 is already newer than 28 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:83: oid 44 version 28 is already newer than 25 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 44 v 28 (ObjNum 66 snap 3 seq_num 66) dirty exists 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:83: left oid 44 (ObjNum 66 snap 3 seq_num 66) 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:87: finishing write tid 3 to trial15342161-42 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:87: finishing write tid 4 to trial15342161-42 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:87: finishing write tid 5 to trial15342161-42 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:87: oid 42 updating version 0 to 18 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:87: oid 42 version 18 is already newer than 18 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:87: oid 42 version 18 is already newer than 18 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:87: oid 42 version 18 is already newer than 18 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 18 (ObjNum 68 snap 3 seq_num 68) dirty exists 2026-02-20T11:59:34.722 INFO:tasks.rados.rados.0.trial153.stdout:87: left oid 42 (ObjNum 68 snap 3 seq_num 68) 2026-02-20T11:59:34.723 INFO:tasks.rados.rados.0.trial153.stdout:97: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2026-02-20T11:59:34.724 INFO:tasks.rados.rados.0.trial153.stdout:97: writing trial15342161-23 from 410104 to 875209 tid 1 2026-02-20T11:59:34.729 INFO:tasks.rados.rados.0.trial153.stdout:97: writing trial15342161-23 from 1563527 to 2146552 tid 2 2026-02-20T11:59:34.733 INFO:tasks.rados.rados.0.trial153.stdout:97: writing trial15342161-23 from 2810926 to 3210104 tid 3 2026-02-20T11:59:34.733 INFO:tasks.rados.rados.0.trial153.stdout:83: done (9 left) 2026-02-20T11:59:34.734 INFO:tasks.rados.rados.0.trial153.stdout:87: done (8 left) 2026-02-20T11:59:34.734 INFO:tasks.rados.rados.0.trial153.stdout:98: read oid 1 snap -1 2026-02-20T11:59:34.734 INFO:tasks.rados.rados.0.trial153.stdout:98: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-20T11:59:34.734 INFO:tasks.rados.rados.0.trial153.stdout:99: write oid 45 current snap is 3 2026-02-20T11:59:34.734 INFO:tasks.rados.rados.0.trial153.stdout:99: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2026-02-20T11:59:34.737 INFO:tasks.rados.rados.0.trial153.stdout:99: writing trial15342161-45 from 426911 to 967265 tid 1 2026-02-20T11:59:34.742 INFO:tasks.rados.rados.0.trial153.stdout:99: writing trial15342161-45 from 1622009 to 2348692 tid 2 2026-02-20T11:59:34.746 INFO:tasks.rados.rados.0.trial153.stdout:99: writing trial15342161-45 from 3121996 to 3226911 tid 3 2026-02-20T11:59:34.746 INFO:tasks.rados.rados.0.trial153.stdout:100: write oid 29 current snap is 3 2026-02-20T11:59:34.746 INFO:tasks.rados.rados.0.trial153.stdout:100: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2026-02-20T11:59:34.750 INFO:tasks.rados.rados.0.trial153.stdout:100: writing trial15342161-29 from 443718 to 1059321 tid 1 2026-02-20T11:59:34.754 INFO:tasks.rados.rados.0.trial153.stdout:100: writing trial15342161-29 from 1564138 to 2034479 tid 2 2026-02-20T11:59:34.758 INFO:tasks.rados.rados.0.trial153.stdout:100: writing trial15342161-29 from 2633066 to 3243718 tid 3 2026-02-20T11:59:34.759 INFO:tasks.rados.rados.0.trial153.stdout:101: delete oid 8 current snap is 3 2026-02-20T11:59:34.759 INFO:tasks.rados.rados.0.trial153.stdout:89: finishing write tid 1 to trial15342161-4 2026-02-20T11:59:34.759 INFO:tasks.rados.rados.0.trial153.stdout:90: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-20T11:59:34.785 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 247843 2026-02-20T11:59:34.805 INFO:tasks.rados.rados.0.trial153.stdout:92: expect (ObjNum 40 snap 0 seq_num 40) 2026-02-20T11:59:34.872 INFO:tasks.rados.rados.0.trial153.stdout:89: finishing write tid 2 to trial15342161-4 2026-02-20T11:59:34.873 INFO:tasks.rados.rados.0.trial153.stdout:90: done (11 left) 2026-02-20T11:59:34.873 INFO:tasks.rados.rados.0.trial153.stdout:92: done (10 left) 2026-02-20T11:59:34.873 INFO:tasks.rados.rados.0.trial153.stdout:101: done (9 left) 2026-02-20T11:59:34.873 INFO:tasks.rados.rados.0.trial153.stdout:102: read oid 26 snap -1 2026-02-20T11:59:34.873 INFO:tasks.rados.rados.0.trial153.stdout:102: expect (ObjNum 25 snap 0 seq_num 25) 2026-02-20T11:59:34.873 INFO:tasks.rados.rados.0.trial153.stdout:103: write oid 32 current snap is 3 2026-02-20T11:59:34.873 INFO:tasks.rados.rados.0.trial153.stdout:89: finishing write tid 3 to trial15342161-4 2026-02-20T11:59:34.873 INFO:tasks.rados.rados.0.trial153.stdout:103: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2026-02-20T11:59:34.877 INFO:tasks.rados.rados.0.trial153.stdout:103: writing trial15342161-32 from 477332 to 1243433 tid 1 2026-02-20T11:59:34.880 INFO:tasks.rados.rados.0.trial153.stdout:103: writing trial15342161-32 from 1681102 to 2155112 tid 2 2026-02-20T11:59:34.885 INFO:tasks.rados.rados.0.trial153.stdout:103: writing trial15342161-32 from 2687912 to 3277332 tid 3 2026-02-20T11:59:34.886 INFO:tasks.rados.rados.0.trial153.stdout:104: read oid 42 snap -1 2026-02-20T11:59:34.886 INFO:tasks.rados.rados.0.trial153.stdout:104: expect (ObjNum 68 snap 3 seq_num 68) 2026-02-20T11:59:34.886 INFO:tasks.rados.rados.0.trial153.stdout:105: snap_create 2026-02-20T11:59:34.886 INFO:tasks.rados.rados.0.trial153.stdout:89: finishing write tid 4 to trial15342161-4 2026-02-20T11:59:34.886 INFO:tasks.rados.rados.0.trial153.stdout:89: finishing write tid 5 to trial15342161-4 2026-02-20T11:59:34.886 INFO:tasks.rados.rados.0.trial153.stdout:89: oid 4 updating version 0 to 18 2026-02-20T11:59:34.886 INFO:tasks.rados.rados.0.trial153.stdout:89: oid 4 version 18 is already newer than 18 2026-02-20T11:59:34.886 INFO:tasks.rados.rados.0.trial153.stdout:89: oid 4 version 18 is already newer than 18 2026-02-20T11:59:34.886 INFO:tasks.rados.rados.0.trial153.stdout:89: oid 4 version 18 is already newer than 18 2026-02-20T11:59:34.886 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 4 v 18 (ObjNum 69 snap 3 seq_num 69) dirty exists 2026-02-20T11:59:34.887 INFO:tasks.rados.rados.0.trial153.stdout:89: left oid 4 (ObjNum 69 snap 3 seq_num 69) 2026-02-20T11:59:34.887 INFO:tasks.rados.rados.0.trial153.stdout:93: expect (ObjNum 30 snap 0 seq_num 30) 2026-02-20T11:59:34.887 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:34.944 INFO:tasks.rados.rados.0.trial153.stdout:91: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T11:59:34.955 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:34.997 INFO:tasks.rados.rados.0.trial153.stdout:94: finishing write tid 1 to trial15342161-3 2026-02-20T11:59:34.998 INFO:tasks.rados.rados.0.trial153.stdout:94: finishing write tid 2 to trial15342161-3 2026-02-20T11:59:34.998 INFO:tasks.rados.rados.0.trial153.stdout:94: finishing write tid 3 to trial15342161-3 2026-02-20T11:59:34.998 INFO:tasks.rados.rados.0.trial153.stdout:94: finishing write tid 4 to trial15342161-3 2026-02-20T11:59:34.998 INFO:tasks.rados.rados.0.trial153.stdout:94: oid 3 updating version 0 to 21 2026-02-20T11:59:34.998 INFO:tasks.rados.rados.0.trial153.stdout:94: oid 3 version 21 is already newer than 21 2026-02-20T11:59:34.998 INFO:tasks.rados.rados.0.trial153.stdout:94: oid 3 version 21 is already newer than 21 2026-02-20T11:59:34.998 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 3 v 21 (ObjNum 70 snap 3 seq_num 70) dirty exists 2026-02-20T11:59:34.998 INFO:tasks.rados.rados.0.trial153.stdout:94: left oid 3 (ObjNum 70 snap 3 seq_num 70) 2026-02-20T11:59:34.998 INFO:tasks.rados.rados.0.trial153.stdout:96: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-20T11:59:35.005 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2084383 2026-02-20T11:59:35.033 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:35.057 INFO:tasks.rados.rados.0.trial153.stdout:97: finishing write tid 1 to trial15342161-23 2026-02-20T11:59:35.057 INFO:tasks.rados.rados.0.trial153.stdout:98: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-20T11:59:35.098 INFO:tasks.rados.rados.0.trial153.stdout:97: finishing write tid 2 to trial15342161-23 2026-02-20T11:59:35.098 INFO:tasks.rados.rados.0.trial153.stdout:97: finishing write tid 3 to trial15342161-23 2026-02-20T11:59:35.098 INFO:tasks.rados.rados.0.trial153.stdout:100: finishing write tid 1 to trial15342161-29 2026-02-20T11:59:35.098 INFO:tasks.rados.rados.0.trial153.stdout:97: finishing write tid 4 to trial15342161-23 2026-02-20T11:59:35.098 INFO:tasks.rados.rados.0.trial153.stdout:97: finishing write tid 5 to trial15342161-23 2026-02-20T11:59:35.098 INFO:tasks.rados.rados.0.trial153.stdout:97: oid 23 updating version 0 to 35 2026-02-20T11:59:35.098 INFO:tasks.rados.rados.0.trial153.stdout:97: oid 23 version 35 is already newer than 35 2026-02-20T11:59:35.098 INFO:tasks.rados.rados.0.trial153.stdout:97: oid 23 version 35 is already newer than 35 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:97: oid 23 version 35 is already newer than 35 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 23 v 35 (ObjNum 72 snap 3 seq_num 72) dirty exists 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:97: left oid 23 (ObjNum 72 snap 3 seq_num 72) 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:100: finishing write tid 2 to trial15342161-29 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:99: finishing write tid 1 to trial15342161-45 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:100: finishing write tid 3 to trial15342161-29 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:99: finishing write tid 2 to trial15342161-45 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:100: finishing write tid 4 to trial15342161-29 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:100: finishing write tid 5 to trial15342161-29 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:100: oid 29 updating version 0 to 30 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:100: oid 29 version 30 is already newer than 30 2026-02-20T11:59:35.099 INFO:tasks.rados.rados.0.trial153.stdout:100: oid 29 version 30 is already newer than 30 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:100: oid 29 version 30 is already newer than 30 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 29 v 30 (ObjNum 74 snap 3 seq_num 74) dirty exists 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:100: left oid 29 (ObjNum 74 snap 3 seq_num 74) 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:99: finishing write tid 3 to trial15342161-45 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:99: finishing write tid 4 to trial15342161-45 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:99: finishing write tid 5 to trial15342161-45 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:99: oid 45 updating version 0 to 15 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:99: oid 45 version 15 is already newer than 15 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:99: oid 45 version 15 is already newer than 15 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:99: oid 45 version 15 is already newer than 15 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 15 (ObjNum 73 snap 3 seq_num 73) dirty exists 2026-02-20T11:59:35.100 INFO:tasks.rados.rados.0.trial153.stdout:99: left oid 45 (ObjNum 73 snap 3 seq_num 73) 2026-02-20T11:59:35.101 INFO:tasks.rados.rados.0.trial153.stdout:102: expect (ObjNum 25 snap 0 seq_num 25) 2026-02-20T11:59:35.136 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:35.157 INFO:tasks.rados.rados.0.trial153.stdout:103: finishing write tid 1 to trial15342161-32 2026-02-20T11:59:35.157 INFO:tasks.rados.rados.0.trial153.stdout:103: finishing write tid 2 to trial15342161-32 2026-02-20T11:59:35.158 INFO:tasks.rados.rados.0.trial153.stdout:103: finishing write tid 3 to trial15342161-32 2026-02-20T11:59:35.158 INFO:tasks.rados.rados.0.trial153.stdout:103: finishing write tid 4 to trial15342161-32 2026-02-20T11:59:35.158 INFO:tasks.rados.rados.0.trial153.stdout:104: expect (ObjNum 68 snap 3 seq_num 68) 2026-02-20T11:59:35.300 INFO:tasks.rados.rados.0.trial153.stdout:103: finishing write tid 5 to trial15342161-32 2026-02-20T11:59:35.300 INFO:tasks.rados.rados.0.trial153.stdout:103: oid 32 updating version 0 to 25 2026-02-20T11:59:35.300 INFO:tasks.rados.rados.0.trial153.stdout:103: oid 32 version 25 is already newer than 25 2026-02-20T11:59:35.300 INFO:tasks.rados.rados.0.trial153.stdout:103: oid 32 version 25 is already newer than 25 2026-02-20T11:59:35.300 INFO:tasks.rados.rados.0.trial153.stdout:103: oid 32 version 25 is already newer than 25 2026-02-20T11:59:35.301 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 32 v 25 (ObjNum 76 snap 3 seq_num 76) dirty exists 2026-02-20T11:59:35.301 INFO:tasks.rados.rados.0.trial153.stdout:103: left oid 32 (ObjNum 76 snap 3 seq_num 76) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:89: done (12 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:91: done (11 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:93: done (10 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:94: done (9 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:96: done (8 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:97: done (7 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:98: done (6 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:99: done (5 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:100: done (4 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:102: done (3 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:103: done (2 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:104: done (1 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:105: done (0 left) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:106: read oid 40 snap 3 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:106: expect (ObjNum 61 snap 2 seq_num 61) 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:107: write oid 37 current snap is 4 2026-02-20T11:59:35.526 INFO:tasks.rados.rados.0.trial153.stdout:107: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2026-02-20T11:59:35.529 INFO:tasks.rados.rados.0.trial153.stdout:107: writing trial15342161-37 from 494139 to 1051842 tid 1 2026-02-20T11:59:35.533 INFO:tasks.rados.rados.0.trial153.stdout:107: writing trial15342161-37 from 1455937 to 2189958 tid 2 2026-02-20T11:59:35.538 INFO:tasks.rados.rados.0.trial153.stdout:107: writing trial15342161-37 from 2948041 to 3294139 tid 3 2026-02-20T11:59:35.538 INFO:tasks.rados.rados.0.trial153.stdout:108: snap_create 2026-02-20T11:59:35.557 INFO:tasks.rados.rados.0.trial153.stdout:106: expect (ObjNum 61 snap 2 seq_num 61) 2026-02-20T11:59:35.557 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 61264 2026-02-20T11:59:35.592 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:35.662 INFO:tasks.rados.rados.0.trial153.stdout:107: finishing write tid 1 to trial15342161-37 2026-02-20T11:59:35.663 INFO:tasks.rados.rados.0.trial153.stdout:107: finishing write tid 2 to trial15342161-37 2026-02-20T11:59:35.663 INFO:tasks.rados.rados.0.trial153.stdout:107: finishing write tid 3 to trial15342161-37 2026-02-20T11:59:35.663 INFO:tasks.rados.rados.0.trial153.stdout:107: finishing write tid 4 to trial15342161-37 2026-02-20T11:59:35.663 INFO:tasks.rados.rados.0.trial153.stdout:107: finishing write tid 5 to trial15342161-37 2026-02-20T11:59:35.663 INFO:tasks.rados.rados.0.trial153.stdout:107: oid 37 updating version 0 to 15 2026-02-20T11:59:35.663 INFO:tasks.rados.rados.0.trial153.stdout:107: oid 37 version 15 is already newer than 15 2026-02-20T11:59:35.663 INFO:tasks.rados.rados.0.trial153.stdout:107: oid 37 version 15 is already newer than 15 2026-02-20T11:59:35.663 INFO:tasks.rados.rados.0.trial153.stdout:107: oid 37 version 15 is already newer than 15 2026-02-20T11:59:35.663 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 15 (ObjNum 77 snap 4 seq_num 77) dirty exists 2026-02-20T11:59:35.663 INFO:tasks.rados.rados.0.trial153.stdout:107: left oid 37 (ObjNum 77 snap 4 seq_num 77) 2026-02-20T11:59:35.818 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~22s 2026-02-20T11:59:35.917 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:59:35.996 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:35.990+0000 7f35fd45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:35.998 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:35.993+0000 7f35fd45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:36.025 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:36.043 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:36.528 INFO:tasks.rados.rados.0.trial153.stdout:106: done (2 left) 2026-02-20T11:59:36.528 INFO:tasks.rados.rados.0.trial153.stdout:107: done (1 left) 2026-02-20T11:59:36.528 INFO:tasks.rados.rados.0.trial153.stdout:108: done (0 left) 2026-02-20T11:59:36.528 INFO:tasks.rados.rados.0.trial153.stdout:109: write (excl) oid 42 current snap is 5 2026-02-20T11:59:36.529 INFO:tasks.rados.rados.0.trial153.stdout:109: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2026-02-20T11:59:36.533 INFO:tasks.rados.rados.0.trial153.stdout:109: writing trial15342161-42 from 510946 to 1143898 tid 1 2026-02-20T11:59:36.538 INFO:tasks.rados.rados.0.trial153.stdout:109: writing trial15342161-42 from 1798066 to 2275745 tid 2 2026-02-20T11:59:36.541 INFO:tasks.rados.rados.0.trial153.stdout:109: writing trial15342161-42 from 2742758 to 3199975 tid 3 2026-02-20T11:59:36.542 INFO:tasks.rados.rados.0.trial153.stdout:110: snap_remove snap 3 2026-02-20T11:59:37.534 INFO:tasks.rados.rados.0.trial153.stdout:110: done (1 left) 2026-02-20T11:59:37.534 INFO:tasks.rados.rados.0.trial153.stdout:111: snap_remove snap 4 2026-02-20T11:59:37.686 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:37.758 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:37.761 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:38.045 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:59:38.123 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:38.117+0000 7f50ff2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:38.126 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:38.121+0000 7f50ff2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:38.153 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:38.169 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:38.537 INFO:tasks.rados.rados.0.trial153.stdout:111: done (1 left) 2026-02-20T11:59:38.538 INFO:tasks.rados.rados.0.trial153.stdout:112: write (excl) oid 28 current snap is 5 2026-02-20T11:59:38.538 INFO:tasks.rados.rados.0.trial153.stdout:109: finishing write tid 1 to trial15342161-42 2026-02-20T11:59:38.538 INFO:tasks.rados.rados.0.trial153.stdout:112: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2026-02-20T11:59:38.543 INFO:tasks.rados.rados.0.trial153.stdout:112: writing trial15342161-28 from 527753 to 1235954 tid 1 2026-02-20T11:59:38.549 INFO:tasks.rados.rados.0.trial153.stdout:112: writing trial15342161-28 from 1856548 to 2594238 tid 2 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:112: writing trial15342161-28 from 3286534 to 3327753 tid 3 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:113: snap_create 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:109: finishing write tid 2 to trial15342161-42 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:109: finishing write tid 3 to trial15342161-42 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:109: finishing write tid 4 to trial15342161-42 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:109: finishing write tid 5 to trial15342161-42 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:109: oid 42 updating version 0 to 23 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:109: oid 42 version 23 is already newer than 23 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:109: oid 42 version 23 is already newer than 20 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:109: oid 42 version 23 is already newer than 23 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 23 (ObjNum 78 snap 5 seq_num 78) dirty exists 2026-02-20T11:59:38.551 INFO:tasks.rados.rados.0.trial153.stdout:109: left oid 42 (ObjNum 78 snap 5 seq_num 78) 2026-02-20T11:59:38.568 INFO:tasks.rados.rados.0.trial153.stdout:112: finishing write tid 1 to trial15342161-28 2026-02-20T11:59:38.580 INFO:tasks.rados.rados.0.trial153.stdout:112: finishing write tid 2 to trial15342161-28 2026-02-20T11:59:38.585 INFO:tasks.rados.rados.0.trial153.stdout:112: finishing write tid 3 to trial15342161-28 2026-02-20T11:59:38.589 INFO:tasks.rados.rados.0.trial153.stdout:112: finishing write tid 4 to trial15342161-28 2026-02-20T11:59:38.591 INFO:tasks.rados.rados.0.trial153.stdout:112: finishing write tid 5 to trial15342161-28 2026-02-20T11:59:38.591 INFO:tasks.rados.rados.0.trial153.stdout:112: oid 28 updating version 0 to 39 2026-02-20T11:59:38.591 INFO:tasks.rados.rados.0.trial153.stdout:112: oid 28 version 39 is already newer than 36 2026-02-20T11:59:38.591 INFO:tasks.rados.rados.0.trial153.stdout:112: oid 28 version 39 is already newer than 39 2026-02-20T11:59:38.591 INFO:tasks.rados.rados.0.trial153.stdout:112: oid 28 version 39 is already newer than 39 2026-02-20T11:59:38.591 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 28 v 39 (ObjNum 79 snap 5 seq_num 79) dirty exists 2026-02-20T11:59:38.591 INFO:tasks.rados.rados.0.trial153.stdout:112: left oid 28 (ObjNum 79 snap 5 seq_num 79) 2026-02-20T11:59:39.541 INFO:tasks.rados.rados.0.trial153.stdout:109: done (2 left) 2026-02-20T11:59:39.541 INFO:tasks.rados.rados.0.trial153.stdout:112: done (1 left) 2026-02-20T11:59:39.541 INFO:tasks.rados.rados.0.trial153.stdout:113: done (0 left) 2026-02-20T11:59:39.541 INFO:tasks.rados.rados.0.trial153.stdout:114: delete oid 14 current snap is 6 2026-02-20T11:59:39.563 INFO:tasks.rados.rados.0.trial153.stdout:114: done (0 left) 2026-02-20T11:59:39.563 INFO:tasks.rados.rados.0.trial153.stdout:115: snap_remove snap 5 2026-02-20T11:59:40.170 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:59:40.246 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:40.240+0000 7f461f8c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:40.248 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:40.244+0000 7f461f8c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:40.283 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:40.299 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:40.544 INFO:tasks.rados.rados.0.trial153.stdout:115: done (0 left) 2026-02-20T11:59:40.545 INFO:tasks.rados.rados.0.trial153.stdout:116: snap_create 2026-02-20T11:59:41.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~28s 2026-02-20T11:59:41.548 INFO:tasks.rados.rados.0.trial153.stdout:116: done (0 left) 2026-02-20T11:59:41.548 INFO:tasks.rados.rados.0.trial153.stdout:117: snap_create 2026-02-20T11:59:42.300 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:59:42.380 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:42.374+0000 7fe9a54cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:42.382 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:42.377+0000 7fe9a54cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:42.408 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:42.424 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:42.552 INFO:tasks.rados.rados.0.trial153.stdout:117: done (0 left) 2026-02-20T11:59:42.552 INFO:tasks.rados.rados.0.trial153.stdout:118: snap_remove snap 6 2026-02-20T11:59:42.762 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:42.835 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:42.838 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:43.558 INFO:tasks.rados.rados.0.trial153.stdout:118: done (0 left) 2026-02-20T11:59:43.559 INFO:tasks.rados.rados.0.trial153.stdout:119: read oid 14 snap 7 2026-02-20T11:59:43.559 INFO:tasks.rados.rados.0.trial153.stdout:119: expect deleted 2026-02-20T11:59:43.559 INFO:tasks.rados.rados.0.trial153.stdout:120: write oid 34 current snap is 8 2026-02-20T11:59:43.559 INFO:tasks.rados.rados.0.trial153.stdout:120: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2026-02-20T11:59:43.562 INFO:tasks.rados.rados.0.trial153.stdout:120: writing trial15342161-34 from 561367 to 1020066 tid 1 2026-02-20T11:59:43.568 INFO:tasks.rados.rados.0.trial153.stdout:120: writing trial15342161-34 from 1573512 to 2314871 tid 2 2026-02-20T11:59:43.572 INFO:tasks.rados.rados.0.trial153.stdout:120: writing trial15342161-34 from 2941380 to 3361367 tid 3 2026-02-20T11:59:43.572 INFO:tasks.rados.rados.0.trial153.stdout:121: read oid 29 snap 7 2026-02-20T11:59:43.572 INFO:tasks.rados.rados.0.trial153.stdout:121: expect (ObjNum 74 snap 3 seq_num 74) 2026-02-20T11:59:43.572 INFO:tasks.rados.rados.0.trial153.stdout:122: write oid 36 current snap is 8 2026-02-20T11:59:43.572 INFO:tasks.rados.rados.0.trial153.stdout:122: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2026-02-20T11:59:43.576 INFO:tasks.rados.rados.0.trial153.stdout:122: writing trial15342161-36 from 578174 to 1112122 tid 1 2026-02-20T11:59:43.579 INFO:tasks.rados.rados.0.trial153.stdout:122: writing trial15342161-36 from 1515641 to 2000658 tid 2 2026-02-20T11:59:43.584 INFO:tasks.rados.rados.0.trial153.stdout:122: writing trial15342161-36 from 2736097 to 3238402 tid 3 2026-02-20T11:59:43.584 INFO:tasks.rados.rados.0.trial153.stdout:123: read oid 46 snap -1 2026-02-20T11:59:43.584 INFO:tasks.rados.rados.0.trial153.stdout:123: expect (ObjNum 45 snap 0 seq_num 45) 2026-02-20T11:59:43.584 INFO:tasks.rados.rados.0.trial153.stdout:124: write oid 28 current snap is 8 2026-02-20T11:59:43.584 INFO:tasks.rados.rados.0.trial153.stdout:124: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2026-02-20T11:59:43.588 INFO:tasks.rados.rados.0.trial153.stdout:124: writing trial15342161-28 from 594981 to 1204178 tid 1 2026-02-20T11:59:43.594 INFO:tasks.rados.rados.0.trial153.stdout:124: writing trial15342161-28 from 1974123 to 2719151 tid 2 2026-02-20T11:59:43.597 INFO:tasks.rados.rados.0.trial153.stdout:124: writing trial15342161-28 from 3279873 to 3394981 tid 3 2026-02-20T11:59:43.597 INFO:tasks.rados.rados.0.trial153.stdout:125: read oid 10 snap -1 2026-02-20T11:59:43.597 INFO:tasks.rados.rados.0.trial153.stdout:125: expect (ObjNum 60 snap 2 seq_num 60) 2026-02-20T11:59:43.597 INFO:tasks.rados.rados.0.trial153.stdout:126: read oid 11 snap -1 2026-02-20T11:59:43.597 INFO:tasks.rados.rados.0.trial153.stdout:126: expect (ObjNum 10 snap 0 seq_num 10) 2026-02-20T11:59:43.598 INFO:tasks.rados.rados.0.trial153.stdout:127: delete oid 41 current snap is 8 2026-02-20T11:59:43.598 INFO:tasks.rados.rados.0.trial153.stdout:121: expect (ObjNum 74 snap 3 seq_num 74) 2026-02-20T11:59:43.730 INFO:tasks.rados.rados.0.trial153.stdout:119: done (8 left) 2026-02-20T11:59:43.730 INFO:tasks.rados.rados.0.trial153.stdout:121: done (7 left) 2026-02-20T11:59:43.730 INFO:tasks.rados.rados.0.trial153.stdout:127: done (6 left) 2026-02-20T11:59:43.730 INFO:tasks.rados.rados.0.trial153.stdout:128: snap_create 2026-02-20T11:59:43.730 INFO:tasks.rados.rados.0.trial153.stdout:123: expect (ObjNum 45 snap 0 seq_num 45) 2026-02-20T11:59:43.730 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 753885 2026-02-20T11:59:43.748 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:43.774 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 844127 2026-02-20T11:59:43.789 INFO:tasks.rados.rados.0.trial153.stdout:126: expect (ObjNum 10 snap 0 seq_num 10) 2026-02-20T11:59:43.828 INFO:tasks.rados.rados.0.trial153.stdout:124: finishing write tid 1 to trial15342161-28 2026-02-20T11:59:43.828 INFO:tasks.rados.rados.0.trial153.stdout:122: finishing write tid 1 to trial15342161-36 2026-02-20T11:59:43.828 INFO:tasks.rados.rados.0.trial153.stdout:122: finishing write tid 2 to trial15342161-36 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:124: finishing write tid 2 to trial15342161-28 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:122: finishing write tid 3 to trial15342161-36 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:122: finishing write tid 4 to trial15342161-36 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:124: finishing write tid 3 to trial15342161-28 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:124: finishing write tid 4 to trial15342161-28 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:124: finishing write tid 5 to trial15342161-28 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:124: oid 28 updating version 0 to 44 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:124: oid 28 version 44 is already newer than 44 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:124: oid 28 version 44 is already newer than 44 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:124: oid 28 version 44 is already newer than 44 2026-02-20T11:59:43.829 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 28 v 44 (ObjNum 83 snap 8 seq_num 83) dirty exists 2026-02-20T11:59:43.830 INFO:tasks.rados.rados.0.trial153.stdout:124: left oid 28 (ObjNum 83 snap 8 seq_num 83) 2026-02-20T11:59:43.830 INFO:tasks.rados.rados.0.trial153.stdout:122: finishing write tid 5 to trial15342161-36 2026-02-20T11:59:43.830 INFO:tasks.rados.rados.0.trial153.stdout:122: oid 36 updating version 0 to 19 2026-02-20T11:59:43.830 INFO:tasks.rados.rados.0.trial153.stdout:122: oid 36 version 19 is already newer than 19 2026-02-20T11:59:43.830 INFO:tasks.rados.rados.0.trial153.stdout:122: oid 36 version 19 is already newer than 19 2026-02-20T11:59:43.830 INFO:tasks.rados.rados.0.trial153.stdout:122: oid 36 version 19 is already newer than 19 2026-02-20T11:59:43.830 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 36 v 19 (ObjNum 82 snap 8 seq_num 82) dirty exists 2026-02-20T11:59:43.830 INFO:tasks.rados.rados.0.trial153.stdout:122: left oid 36 (ObjNum 82 snap 8 seq_num 82) 2026-02-20T11:59:43.830 INFO:tasks.rados.rados.0.trial153.stdout:125: expect (ObjNum 60 snap 2 seq_num 60) 2026-02-20T11:59:43.935 INFO:tasks.rados.rados.0.trial153.stdout:120: finishing write tid 1 to trial15342161-34 2026-02-20T11:59:43.935 INFO:tasks.rados.rados.0.trial153.stdout:120: finishing write tid 2 to trial15342161-34 2026-02-20T11:59:43.935 INFO:tasks.rados.rados.0.trial153.stdout:120: finishing write tid 3 to trial15342161-34 2026-02-20T11:59:43.935 INFO:tasks.rados.rados.0.trial153.stdout:120: finishing write tid 4 to trial15342161-34 2026-02-20T11:59:43.936 INFO:tasks.rados.rados.0.trial153.stdout:120: finishing write tid 5 to trial15342161-34 2026-02-20T11:59:43.936 INFO:tasks.rados.rados.0.trial153.stdout:120: oid 34 updating version 0 to 33 2026-02-20T11:59:43.936 INFO:tasks.rados.rados.0.trial153.stdout:120: oid 34 version 33 is already newer than 33 2026-02-20T11:59:43.936 INFO:tasks.rados.rados.0.trial153.stdout:120: oid 34 version 33 is already newer than 33 2026-02-20T11:59:43.936 INFO:tasks.rados.rados.0.trial153.stdout:120: oid 34 version 33 is already newer than 33 2026-02-20T11:59:43.936 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 33 (ObjNum 81 snap 8 seq_num 81) dirty exists 2026-02-20T11:59:43.936 INFO:tasks.rados.rados.0.trial153.stdout:120: left oid 34 (ObjNum 81 snap 8 seq_num 81) 2026-02-20T11:59:44.425 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:59:44.504 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:44.498+0000 7faa8cb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:44.507 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:44.502+0000 7faa8cb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:44.535 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:44.552 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:44.560 INFO:tasks.rados.rados.0.trial153.stdout:120: done (6 left) 2026-02-20T11:59:44.560 INFO:tasks.rados.rados.0.trial153.stdout:122: done (5 left) 2026-02-20T11:59:44.560 INFO:tasks.rados.rados.0.trial153.stdout:123: done (4 left) 2026-02-20T11:59:44.560 INFO:tasks.rados.rados.0.trial153.stdout:124: done (3 left) 2026-02-20T11:59:44.560 INFO:tasks.rados.rados.0.trial153.stdout:125: done (2 left) 2026-02-20T11:59:44.560 INFO:tasks.rados.rados.0.trial153.stdout:126: done (1 left) 2026-02-20T11:59:44.560 INFO:tasks.rados.rados.0.trial153.stdout:128: done (0 left) 2026-02-20T11:59:44.560 INFO:tasks.rados.rados.0.trial153.stdout:129: delete oid 48 current snap is 9 2026-02-20T11:59:44.596 INFO:tasks.rados.rados.0.trial153.stdout:129: done (0 left) 2026-02-20T11:59:44.596 INFO:tasks.rados.rados.0.trial153.stdout:130: read oid 45 snap 7 2026-02-20T11:59:44.596 INFO:tasks.rados.rados.0.trial153.stdout:130: expect (ObjNum 73 snap 3 seq_num 73) 2026-02-20T11:59:44.597 INFO:tasks.rados.rados.0.trial153.stdout:131: write oid 2 current snap is 9 2026-02-20T11:59:44.597 INFO:tasks.rados.rados.0.trial153.stdout:131: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2026-02-20T11:59:44.600 INFO:tasks.rados.rados.0.trial153.stdout:131: writing trial15342161-2 from 645402 to 1196699 tid 1 2026-02-20T11:59:44.605 INFO:tasks.rados.rados.0.trial153.stdout:131: writing trial15342161-2 from 1749569 to 2241924 tid 2 2026-02-20T11:59:44.610 INFO:tasks.rados.rados.0.trial153.stdout:131: writing trial15342161-2 from 2845789 to 3445402 tid 3 2026-02-20T11:59:44.610 INFO:tasks.rados.rados.0.trial153.stdout:132: snap_create 2026-02-20T11:59:44.630 INFO:tasks.rados.rados.0.trial153.stdout:130: expect (ObjNum 73 snap 3 seq_num 73) 2026-02-20T11:59:44.689 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1414407 2026-02-20T11:59:44.738 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2484348 2026-02-20T11:59:44.756 INFO:tasks.rados.rados.0.trial153.stdout:131: finishing write tid 1 to trial15342161-2 2026-02-20T11:59:44.757 INFO:tasks.rados.rados.0.trial153.stdout:131: finishing write tid 2 to trial15342161-2 2026-02-20T11:59:44.757 INFO:tasks.rados.rados.0.trial153.stdout:131: finishing write tid 3 to trial15342161-2 2026-02-20T11:59:44.757 INFO:tasks.rados.rados.0.trial153.stdout:131: finishing write tid 4 to trial15342161-2 2026-02-20T11:59:44.757 INFO:tasks.rados.rados.0.trial153.stdout:131: finishing write tid 5 to trial15342161-2 2026-02-20T11:59:44.757 INFO:tasks.rados.rados.0.trial153.stdout:131: oid 2 updating version 0 to 35 2026-02-20T11:59:44.757 INFO:tasks.rados.rados.0.trial153.stdout:131: oid 2 version 35 is already newer than 35 2026-02-20T11:59:44.757 INFO:tasks.rados.rados.0.trial153.stdout:131: oid 2 version 35 is already newer than 35 2026-02-20T11:59:44.757 INFO:tasks.rados.rados.0.trial153.stdout:131: oid 2 version 35 is already newer than 35 2026-02-20T11:59:44.757 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 2 v 35 (ObjNum 86 snap 9 seq_num 86) dirty exists 2026-02-20T11:59:44.758 INFO:tasks.rados.rados.0.trial153.stdout:131: left oid 2 (ObjNum 86 snap 9 seq_num 86) 2026-02-20T11:59:45.565 INFO:tasks.rados.rados.0.trial153.stdout:130: done (2 left) 2026-02-20T11:59:45.565 INFO:tasks.rados.rados.0.trial153.stdout:131: done (1 left) 2026-02-20T11:59:45.565 INFO:tasks.rados.rados.0.trial153.stdout:132: done (0 left) 2026-02-20T11:59:45.565 INFO:tasks.rados.rados.0.trial153.stdout:133: snap_remove snap 8 2026-02-20T11:59:46.553 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:59:46.564 INFO:tasks.rados.rados.0.trial153.stdout:133: done (0 left) 2026-02-20T11:59:46.564 INFO:tasks.rados.rados.0.trial153.stdout:134: write (excl) oid 26 current snap is 10 2026-02-20T11:59:46.564 INFO:tasks.rados.rados.0.trial153.stdout:134: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2026-02-20T11:59:46.569 INFO:tasks.rados.rados.0.trial153.stdout:134: writing trial15342161-26 from 662209 to 1288755 tid 1 2026-02-20T11:59:46.574 INFO:tasks.rados.rados.0.trial153.stdout:134: writing trial15342161-26 from 1808051 to 2444064 tid 2 2026-02-20T11:59:46.578 INFO:tasks.rados.rados.0.trial153.stdout:134: writing trial15342161-26 from 2873212 to 3462209 tid 3 2026-02-20T11:59:46.578 INFO:tasks.rados.rados.0.trial153.stdout:135: snap_remove snap 7 2026-02-20T11:59:46.640 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:46.634+0000 7f9465e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:46.642 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:46.638+0000 7f9465e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:46.670 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:46.686 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:46.827 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~33s 2026-02-20T11:59:47.571 INFO:tasks.rados.rados.0.trial153.stdout:135: done (1 left) 2026-02-20T11:59:47.571 INFO:tasks.rados.rados.0.trial153.stdout:136: read oid 7 snap -1 2026-02-20T11:59:47.571 INFO:tasks.rados.rados.0.trial153.stdout:136: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:59:47.571 INFO:tasks.rados.rados.0.trial153.stdout:137: delete oid 3 current snap is 10 2026-02-20T11:59:47.571 INFO:tasks.rados.rados.0.trial153.stdout:134: finishing write tid 1 to trial15342161-26 2026-02-20T11:59:47.571 INFO:tasks.rados.rados.0.trial153.stdout:134: finishing write tid 2 to trial15342161-26 2026-02-20T11:59:47.571 INFO:tasks.rados.rados.0.trial153.stdout:134: finishing write tid 3 to trial15342161-26 2026-02-20T11:59:47.572 INFO:tasks.rados.rados.0.trial153.stdout:134: finishing write tid 4 to trial15342161-26 2026-02-20T11:59:47.572 INFO:tasks.rados.rados.0.trial153.stdout:134: finishing write tid 5 to trial15342161-26 2026-02-20T11:59:47.572 INFO:tasks.rados.rados.0.trial153.stdout:134: oid 26 updating version 0 to 18 2026-02-20T11:59:47.572 INFO:tasks.rados.rados.0.trial153.stdout:134: oid 26 version 18 is already newer than 18 2026-02-20T11:59:47.572 INFO:tasks.rados.rados.0.trial153.stdout:134: oid 26 version 18 is already newer than 18 2026-02-20T11:59:47.572 INFO:tasks.rados.rados.0.trial153.stdout:134: oid 26 version 18 is already newer than 15 2026-02-20T11:59:47.572 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 26 v 18 (ObjNum 87 snap 10 seq_num 87) dirty exists 2026-02-20T11:59:47.572 INFO:tasks.rados.rados.0.trial153.stdout:134: left oid 26 (ObjNum 87 snap 10 seq_num 87) 2026-02-20T11:59:47.600 INFO:tasks.rados.rados.0.trial153.stdout:136: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T11:59:47.610 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1840934 2026-02-20T11:59:47.641 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2320821 2026-02-20T11:59:47.687 INFO:tasks.rados.rados.0.trial153.stdout:134: done (2 left) 2026-02-20T11:59:47.687 INFO:tasks.rados.rados.0.trial153.stdout:136: done (1 left) 2026-02-20T11:59:47.687 INFO:tasks.rados.rados.0.trial153.stdout:137: done (0 left) 2026-02-20T11:59:47.688 INFO:tasks.rados.rados.0.trial153.stdout:138: snap_remove snap 9 2026-02-20T11:59:47.839 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:47.910 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:47.913 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:48.574 INFO:tasks.rados.rados.0.trial153.stdout:138: done (0 left) 2026-02-20T11:59:48.574 INFO:tasks.rados.rados.0.trial153.stdout:139: write (excl) oid 16 current snap is 10 2026-02-20T11:59:48.574 INFO:tasks.rados.rados.0.trial153.stdout:139: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2026-02-20T11:59:48.580 INFO:tasks.rados.rados.0.trial153.stdout:139: writing trial15342161-16 from 695823 to 1472867 tid 1 2026-02-20T11:59:48.585 INFO:tasks.rados.rados.0.trial153.stdout:139: writing trial15342161-16 from 1925015 to 2564697 tid 2 2026-02-20T11:59:48.588 INFO:tasks.rados.rados.0.trial153.stdout:139: writing trial15342161-16 from 3328058 to 3495823 tid 3 2026-02-20T11:59:48.588 INFO:tasks.rados.rados.0.trial153.stdout:140: write (excl) oid 25 current snap is 10 2026-02-20T11:59:48.588 INFO:tasks.rados.rados.0.trial153.stdout:140: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2026-02-20T11:59:48.593 INFO:tasks.rados.rados.0.trial153.stdout:140: writing trial15342161-25 from 712630 to 1164923 tid 1 2026-02-20T11:59:48.596 INFO:tasks.rados.rados.0.trial153.stdout:140: writing trial15342161-25 from 1583497 to 2083190 tid 2 2026-02-20T11:59:48.601 INFO:tasks.rados.rados.0.trial153.stdout:140: writing trial15342161-25 from 2555481 to 2980668 tid 3 2026-02-20T11:59:48.601 INFO:tasks.rados.rados.0.trial153.stdout:141: write (excl) oid 46 current snap is 10 2026-02-20T11:59:48.601 INFO:tasks.rados.rados.0.trial153.stdout:141: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2026-02-20T11:59:48.605 INFO:tasks.rados.rados.0.trial153.stdout:141: writing trial15342161-46 from 729437 to 1256979 tid 1 2026-02-20T11:59:48.610 INFO:tasks.rados.rados.0.trial153.stdout:141: writing trial15342161-46 from 1925626 to 2568977 tid 2 2026-02-20T11:59:48.614 INFO:tasks.rados.rados.0.trial153.stdout:141: writing trial15342161-46 from 3266551 to 3529437 tid 3 2026-02-20T11:59:48.615 INFO:tasks.rados.rados.0.trial153.stdout:142: read oid 43 snap -1 2026-02-20T11:59:48.615 INFO:tasks.rados.rados.0.trial153.stdout:142: expect (ObjNum 42 snap 0 seq_num 42) 2026-02-20T11:59:48.615 INFO:tasks.rados.rados.0.trial153.stdout:143: delete oid 20 current snap is 10 2026-02-20T11:59:48.615 INFO:tasks.rados.rados.0.trial153.stdout:139: finishing write tid 1 to trial15342161-16 2026-02-20T11:59:48.618 INFO:tasks.rados.rados.0.trial153.stdout:140: finishing write tid 1 to trial15342161-25 2026-02-20T11:59:48.629 INFO:tasks.rados.rados.0.trial153.stdout:141: finishing write tid 1 to trial15342161-46 2026-02-20T11:59:48.634 INFO:tasks.rados.rados.0.trial153.stdout:142: expect (ObjNum 42 snap 0 seq_num 42) 2026-02-20T11:59:48.668 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:48.687 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:59:48.692 INFO:tasks.rados.rados.0.trial153.stdout:142: done (4 left) 2026-02-20T11:59:48.692 INFO:tasks.rados.rados.0.trial153.stdout:143: done (3 left) 2026-02-20T11:59:48.692 INFO:tasks.rados.rados.0.trial153.stdout:144: snap_create 2026-02-20T11:59:48.692 INFO:tasks.rados.rados.0.trial153.stdout:141: finishing write tid 2 to trial15342161-46 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:141: finishing write tid 3 to trial15342161-46 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:141: finishing write tid 4 to trial15342161-46 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:141: finishing write tid 5 to trial15342161-46 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:141: oid 46 updating version 0 to 10 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:141: oid 46 version 10 is already newer than 10 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:141: oid 46 version 10 is already newer than 10 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:141: oid 46 version 10 is already newer than 7 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 46 v 10 (ObjNum 91 snap 10 seq_num 91) dirty exists 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:141: left oid 46 (ObjNum 91 snap 10 seq_num 91) 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:140: finishing write tid 2 to trial15342161-25 2026-02-20T11:59:48.693 INFO:tasks.rados.rados.0.trial153.stdout:140: finishing write tid 3 to trial15342161-25 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:139: finishing write tid 2 to trial15342161-16 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:140: finishing write tid 4 to trial15342161-25 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:140: finishing write tid 5 to trial15342161-25 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:140: oid 25 updating version 0 to 10 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:140: oid 25 updating version 10 to 13 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:140: oid 25 version 13 is already newer than 13 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:140: oid 25 version 13 is already newer than 13 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 25 v 13 (ObjNum 90 snap 10 seq_num 90) dirty exists 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:140: left oid 25 (ObjNum 90 snap 10 seq_num 90) 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:139: finishing write tid 3 to trial15342161-16 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:139: finishing write tid 4 to trial15342161-16 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:139: finishing write tid 5 to trial15342161-16 2026-02-20T11:59:48.694 INFO:tasks.rados.rados.0.trial153.stdout:139: oid 16 updating version 0 to 37 2026-02-20T11:59:48.695 INFO:tasks.rados.rados.0.trial153.stdout:139: oid 16 version 37 is already newer than 37 2026-02-20T11:59:48.695 INFO:tasks.rados.rados.0.trial153.stdout:139: oid 16 version 37 is already newer than 37 2026-02-20T11:59:48.695 INFO:tasks.rados.rados.0.trial153.stdout:139: oid 16 version 37 is already newer than 34 2026-02-20T11:59:48.695 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 16 v 37 (ObjNum 89 snap 10 seq_num 89) dirty exists 2026-02-20T11:59:48.695 INFO:tasks.rados.rados.0.trial153.stdout:139: left oid 16 (ObjNum 89 snap 10 seq_num 89) 2026-02-20T11:59:48.762 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:48.756+0000 7f25c6ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:48.765 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:48.760+0000 7f25c6ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:48.792 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:48.808 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:49.579 INFO:tasks.rados.rados.0.trial153.stdout:139: done (3 left) 2026-02-20T11:59:49.579 INFO:tasks.rados.rados.0.trial153.stdout:140: done (2 left) 2026-02-20T11:59:49.579 INFO:tasks.rados.rados.0.trial153.stdout:141: done (1 left) 2026-02-20T11:59:49.579 INFO:tasks.rados.rados.0.trial153.stdout:144: done (0 left) 2026-02-20T11:59:49.579 INFO:tasks.rados.rados.0.trial153.stdout:145: read oid 1 snap -1 2026-02-20T11:59:49.580 INFO:tasks.rados.rados.0.trial153.stdout:145: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-20T11:59:49.580 INFO:tasks.rados.rados.0.trial153.stdout:146: snap_remove snap 10 2026-02-20T11:59:50.581 INFO:tasks.rados.rados.0.trial153.stdout:146: done (1 left) 2026-02-20T11:59:50.581 INFO:tasks.rados.rados.0.trial153.stdout:147: delete oid 6 current snap is 11 2026-02-20T11:59:50.581 INFO:tasks.rados.rados.0.trial153.stdout:145: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-20T11:59:50.581 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 328614 2026-02-20T11:59:50.596 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:50.626 INFO:tasks.rados.rados.0.trial153.stdout:145: done (1 left) 2026-02-20T11:59:50.626 INFO:tasks.rados.rados.0.trial153.stdout:147: done (0 left) 2026-02-20T11:59:50.626 INFO:tasks.rados.rados.0.trial153.stdout:148: write oid 41 current snap is 11 2026-02-20T11:59:50.626 INFO:tasks.rados.rados.0.trial153.stdout:148: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2026-02-20T11:59:50.630 INFO:tasks.rados.rados.0.trial153.stdout:148: writing trial15342161-41 from 779858 to 1249500 tid 1 2026-02-20T11:59:50.634 INFO:tasks.rados.rados.0.trial153.stdout:148: writing trial15342161-41 from 1817425 to 2324456 tid 2 2026-02-20T11:59:50.638 INFO:tasks.rados.rados.0.trial153.stdout:148: writing trial15342161-41 from 2781526 to 3368154 tid 3 2026-02-20T11:59:50.638 INFO:tasks.rados.rados.0.trial153.stdout:149: write oid 20 current snap is 11 2026-02-20T11:59:50.639 INFO:tasks.rados.rados.0.trial153.stdout:149: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2026-02-20T11:59:50.644 INFO:tasks.rados.rados.0.trial153.stdout:149: writing trial15342161-20 from 796665 to 1341556 tid 1 2026-02-20T11:59:50.648 INFO:tasks.rados.rados.0.trial153.stdout:149: writing trial15342161-20 from 1759554 to 2410243 tid 2 2026-02-20T11:59:50.653 INFO:tasks.rados.rados.0.trial153.stdout:149: writing trial15342161-20 from 2976243 to 3596665 tid 3 2026-02-20T11:59:50.653 INFO:tasks.rados.rados.0.trial153.stdout:150: delete oid 30 current snap is 11 2026-02-20T11:59:50.658 INFO:tasks.rados.rados.0.trial153.stdout:148: finishing write tid 1 to trial15342161-41 2026-02-20T11:59:50.663 INFO:tasks.rados.rados.0.trial153.stdout:149: finishing write tid 1 to trial15342161-20 2026-02-20T11:59:50.667 INFO:tasks.rados.rados.0.trial153.stdout:148: finishing write tid 2 to trial15342161-41 2026-02-20T11:59:50.674 INFO:tasks.rados.rados.0.trial153.stdout:148: finishing write tid 3 to trial15342161-41 2026-02-20T11:59:50.676 INFO:tasks.rados.rados.0.trial153.stdout:149: finishing write tid 2 to trial15342161-20 2026-02-20T11:59:50.680 INFO:tasks.rados.rados.0.trial153.stdout:148: finishing write tid 4 to trial15342161-41 2026-02-20T11:59:50.681 INFO:tasks.rados.rados.0.trial153.stdout:148: finishing write tid 5 to trial15342161-41 2026-02-20T11:59:50.681 INFO:tasks.rados.rados.0.trial153.stdout:148: oid 41 updating version 0 to 29 2026-02-20T11:59:50.681 INFO:tasks.rados.rados.0.trial153.stdout:148: oid 41 version 29 is already newer than 29 2026-02-20T11:59:50.681 INFO:tasks.rados.rados.0.trial153.stdout:148: oid 41 version 29 is already newer than 29 2026-02-20T11:59:50.681 INFO:tasks.rados.rados.0.trial153.stdout:148: oid 41 version 29 is already newer than 29 2026-02-20T11:59:50.681 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 41 v 29 (ObjNum 94 snap 11 seq_num 94) dirty exists 2026-02-20T11:59:50.682 INFO:tasks.rados.rados.0.trial153.stdout:148: left oid 41 (ObjNum 94 snap 11 seq_num 94) 2026-02-20T11:59:50.687 INFO:tasks.rados.rados.0.trial153.stdout:149: finishing write tid 3 to trial15342161-20 2026-02-20T11:59:50.693 INFO:tasks.rados.rados.0.trial153.stdout:149: finishing write tid 4 to trial15342161-20 2026-02-20T11:59:50.695 INFO:tasks.rados.rados.0.trial153.stdout:149: finishing write tid 5 to trial15342161-20 2026-02-20T11:59:50.695 INFO:tasks.rados.rados.0.trial153.stdout:149: oid 20 updating version 0 to 18 2026-02-20T11:59:50.695 INFO:tasks.rados.rados.0.trial153.stdout:149: oid 20 version 18 is already newer than 18 2026-02-20T11:59:50.695 INFO:tasks.rados.rados.0.trial153.stdout:149: oid 20 version 18 is already newer than 18 2026-02-20T11:59:50.695 INFO:tasks.rados.rados.0.trial153.stdout:149: oid 20 version 18 is already newer than 18 2026-02-20T11:59:50.695 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 20 v 18 (ObjNum 95 snap 11 seq_num 95) dirty exists 2026-02-20T11:59:50.695 INFO:tasks.rados.rados.0.trial153.stdout:149: left oid 20 (ObjNum 95 snap 11 seq_num 95) 2026-02-20T11:59:50.699 INFO:tasks.rados.rados.0.trial153.stdout:148: done (2 left) 2026-02-20T11:59:50.699 INFO:tasks.rados.rados.0.trial153.stdout:149: done (1 left) 2026-02-20T11:59:50.699 INFO:tasks.rados.rados.0.trial153.stdout:150: done (0 left) 2026-02-20T11:59:50.699 INFO:tasks.rados.rados.0.trial153.stdout:151: write (excl) oid 45 current snap is 11 2026-02-20T11:59:50.699 INFO:tasks.rados.rados.0.trial153.stdout:151: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2026-02-20T11:59:50.703 INFO:tasks.rados.rados.0.trial153.stdout:151: writing trial15342161-45 from 430279 to 1125668 tid 1 2026-02-20T11:59:50.709 INFO:tasks.rados.rados.0.trial153.stdout:151: writing trial15342161-45 from 1876518 to 2530876 tid 2 2026-02-20T11:59:50.713 INFO:tasks.rados.rados.0.trial153.stdout:151: writing trial15342161-45 from 3031089 to 3567886 tid 3 2026-02-20T11:59:50.713 INFO:tasks.rados.rados.0.trial153.stdout:152: write (excl) oid 4 current snap is 11 2026-02-20T11:59:50.713 INFO:tasks.rados.rados.0.trial153.stdout:152: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2026-02-20T11:59:50.717 INFO:tasks.rados.rados.0.trial153.stdout:152: writing trial15342161-4 from 447086 to 1217724 tid 1 2026-02-20T11:59:50.723 INFO:tasks.rados.rados.0.trial153.stdout:152: writing trial15342161-4 from 1935000 to 2733016 tid 2 2026-02-20T11:59:50.727 INFO:tasks.rados.rados.0.trial153.stdout:152: writing trial15342161-4 from 3458512 to 3647086 tid 3 2026-02-20T11:59:50.727 INFO:tasks.rados.rados.0.trial153.stdout:153: delete oid 30 current snap is 11 2026-02-20T11:59:50.727 INFO:tasks.rados.rados.0.trial153.stdout:151: finishing write tid 1 to trial15342161-45 2026-02-20T11:59:50.736 INFO:tasks.rados.rados.0.trial153.stdout:151: finishing write tid 2 to trial15342161-45 2026-02-20T11:59:50.737 INFO:tasks.rados.rados.0.trial153.stdout:153: done (2 left) 2026-02-20T11:59:50.737 INFO:tasks.rados.rados.0.trial153.stdout:154: write (excl) oid 3 current snap is 11 2026-02-20T11:59:50.737 INFO:tasks.rados.rados.0.trial153.stdout:154: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2026-02-20T11:59:50.740 INFO:tasks.rados.rados.0.trial153.stdout:154: writing trial15342161-3 from 480700 to 1118189 tid 1 2026-02-20T11:59:50.744 INFO:tasks.rados.rados.0.trial153.stdout:154: writing trial15342161-3 from 1651964 to 2053649 tid 2 2026-02-20T11:59:50.750 INFO:tasks.rados.rados.0.trial153.stdout:154: writing trial15342161-3 from 2713358 to 3483971 tid 3 2026-02-20T11:59:50.750 INFO:tasks.rados.rados.0.trial153.stdout:155: write oid 26 current snap is 11 2026-02-20T11:59:50.750 INFO:tasks.rados.rados.0.trial153.stdout:152: finishing write tid 1 to trial15342161-4 2026-02-20T11:59:50.751 INFO:tasks.rados.rados.0.trial153.stdout:151: finishing write tid 3 to trial15342161-45 2026-02-20T11:59:50.751 INFO:tasks.rados.rados.0.trial153.stdout:155: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2026-02-20T11:59:50.754 INFO:tasks.rados.rados.0.trial153.stdout:155: writing trial15342161-26 from 497507 to 1210245 tid 1 2026-02-20T11:59:50.759 INFO:tasks.rados.rados.0.trial153.stdout:155: writing trial15342161-26 from 1710446 to 2372142 tid 2 2026-02-20T11:59:50.765 INFO:tasks.rados.rados.0.trial153.stdout:155: writing trial15342161-26 from 3140781 to 3697507 tid 3 2026-02-20T11:59:50.765 INFO:tasks.rados.rados.0.trial153.stdout:156: snap_create 2026-02-20T11:59:50.765 INFO:tasks.rados.rados.0.trial153.stdout:151: finishing write tid 4 to trial15342161-45 2026-02-20T11:59:50.765 INFO:tasks.rados.rados.0.trial153.stdout:152: finishing write tid 2 to trial15342161-4 2026-02-20T11:59:50.765 INFO:tasks.rados.rados.0.trial153.stdout:151: finishing write tid 5 to trial15342161-45 2026-02-20T11:59:50.765 INFO:tasks.rados.rados.0.trial153.stdout:151: oid 45 updating version 0 to 19 2026-02-20T11:59:50.766 INFO:tasks.rados.rados.0.trial153.stdout:151: oid 45 version 19 is already newer than 19 2026-02-20T11:59:50.766 INFO:tasks.rados.rados.0.trial153.stdout:151: oid 45 version 19 is already newer than 16 2026-02-20T11:59:50.766 INFO:tasks.rados.rados.0.trial153.stdout:151: oid 45 version 19 is already newer than 19 2026-02-20T11:59:50.766 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 19 (ObjNum 97 snap 11 seq_num 97) dirty exists 2026-02-20T11:59:50.766 INFO:tasks.rados.rados.0.trial153.stdout:151: left oid 45 (ObjNum 97 snap 11 seq_num 97) 2026-02-20T11:59:50.766 INFO:tasks.rados.rados.0.trial153.stdout:152: finishing write tid 3 to trial15342161-4 2026-02-20T11:59:50.775 INFO:tasks.rados.rados.0.trial153.stdout:152: finishing write tid 4 to trial15342161-4 2026-02-20T11:59:50.777 INFO:tasks.rados.rados.0.trial153.stdout:152: finishing write tid 5 to trial15342161-4 2026-02-20T11:59:50.777 INFO:tasks.rados.rados.0.trial153.stdout:152: oid 4 updating version 0 to 31 2026-02-20T11:59:50.777 INFO:tasks.rados.rados.0.trial153.stdout:152: oid 4 version 31 is already newer than 28 2026-02-20T11:59:50.778 INFO:tasks.rados.rados.0.trial153.stdout:152: oid 4 version 31 is already newer than 31 2026-02-20T11:59:50.778 INFO:tasks.rados.rados.0.trial153.stdout:152: oid 4 version 31 is already newer than 31 2026-02-20T11:59:50.778 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 4 v 31 (ObjNum 98 snap 11 seq_num 98) dirty exists 2026-02-20T11:59:50.778 INFO:tasks.rados.rados.0.trial153.stdout:152: left oid 4 (ObjNum 98 snap 11 seq_num 98) 2026-02-20T11:59:50.786 INFO:tasks.rados.rados.0.trial153.stdout:154: finishing write tid 1 to trial15342161-3 2026-02-20T11:59:50.794 INFO:tasks.rados.rados.0.trial153.stdout:155: finishing write tid 1 to trial15342161-26 2026-02-20T11:59:50.805 INFO:tasks.rados.rados.0.trial153.stdout:154: finishing write tid 2 to trial15342161-3 2026-02-20T11:59:50.809 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:59:50.814 INFO:tasks.rados.rados.0.trial153.stdout:154: finishing write tid 3 to trial15342161-3 2026-02-20T11:59:50.823 INFO:tasks.rados.rados.0.trial153.stdout:154: finishing write tid 4 to trial15342161-3 2026-02-20T11:59:50.824 INFO:tasks.rados.rados.0.trial153.stdout:154: finishing write tid 5 to trial15342161-3 2026-02-20T11:59:50.824 INFO:tasks.rados.rados.0.trial153.stdout:154: oid 3 updating version 0 to 32 2026-02-20T11:59:50.824 INFO:tasks.rados.rados.0.trial153.stdout:154: oid 3 updating version 32 to 35 2026-02-20T11:59:50.825 INFO:tasks.rados.rados.0.trial153.stdout:154: oid 3 version 35 is already newer than 35 2026-02-20T11:59:50.825 INFO:tasks.rados.rados.0.trial153.stdout:154: oid 3 version 35 is already newer than 35 2026-02-20T11:59:50.825 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 3 v 35 (ObjNum 100 snap 11 seq_num 100) dirty exists 2026-02-20T11:59:50.825 INFO:tasks.rados.rados.0.trial153.stdout:154: left oid 3 (ObjNum 100 snap 11 seq_num 100) 2026-02-20T11:59:50.828 INFO:tasks.rados.rados.0.trial153.stdout:155: finishing write tid 2 to trial15342161-26 2026-02-20T11:59:50.834 INFO:tasks.rados.rados.0.trial153.stdout:155: finishing write tid 3 to trial15342161-26 2026-02-20T11:59:50.840 INFO:tasks.rados.rados.0.trial153.stdout:155: finishing write tid 4 to trial15342161-26 2026-02-20T11:59:50.840 INFO:tasks.rados.rados.0.trial153.stdout:155: finishing write tid 5 to trial15342161-26 2026-02-20T11:59:50.841 INFO:tasks.rados.rados.0.trial153.stdout:155: oid 26 updating version 0 to 26 2026-02-20T11:59:50.841 INFO:tasks.rados.rados.0.trial153.stdout:155: oid 26 version 26 is already newer than 26 2026-02-20T11:59:50.841 INFO:tasks.rados.rados.0.trial153.stdout:155: oid 26 version 26 is already newer than 26 2026-02-20T11:59:50.841 INFO:tasks.rados.rados.0.trial153.stdout:155: oid 26 version 26 is already newer than 26 2026-02-20T11:59:50.841 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 26 v 26 (ObjNum 101 snap 11 seq_num 101) dirty exists 2026-02-20T11:59:50.841 INFO:tasks.rados.rados.0.trial153.stdout:155: left oid 26 (ObjNum 101 snap 11 seq_num 101) 2026-02-20T11:59:50.885 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:50.880+0000 7fe32efa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:50.888 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:50.883+0000 7fe32efa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:50.913 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:50.930 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:51.586 INFO:tasks.rados.rados.0.trial153.stdout:151: done (4 left) 2026-02-20T11:59:51.586 INFO:tasks.rados.rados.0.trial153.stdout:152: done (3 left) 2026-02-20T11:59:51.586 INFO:tasks.rados.rados.0.trial153.stdout:154: done (2 left) 2026-02-20T11:59:51.587 INFO:tasks.rados.rados.0.trial153.stdout:155: done (1 left) 2026-02-20T11:59:51.587 INFO:tasks.rados.rados.0.trial153.stdout:156: done (0 left) 2026-02-20T11:59:51.587 INFO:tasks.rados.rados.0.trial153.stdout:157: snap_create 2026-02-20T11:59:52.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~39s 2026-02-20T11:59:52.591 INFO:tasks.rados.rados.0.trial153.stdout:157: done (0 left) 2026-02-20T11:59:52.591 INFO:tasks.rados.rados.0.trial153.stdout:158: read oid 2 snap -1 2026-02-20T11:59:52.591 INFO:tasks.rados.rados.0.trial153.stdout:158: expect (ObjNum 86 snap 9 seq_num 86) 2026-02-20T11:59:52.591 INFO:tasks.rados.rados.0.trial153.stdout:159: snap_remove snap 12 2026-02-20T11:59:52.914 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:52.931 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:59:52.987 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:52.990 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:52.990 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:53.008 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:53.003+0000 7f651d88b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:53.010 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:53.006+0000 7f651d88b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:53.037 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:53.052 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:53.052 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:53.591 INFO:tasks.rados.rados.0.trial153.stdout:159: done (1 left) 2026-02-20T11:59:53.591 INFO:tasks.rados.rados.0.trial153.stdout:160: write (excl) oid 40 current snap is 13 2026-02-20T11:59:53.592 INFO:tasks.rados.rados.0.trial153.stdout:160: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2026-02-20T11:59:53.597 INFO:tasks.rados.rados.0.trial153.stdout:160: writing trial15342161-40 from 514314 to 1302301 tid 1 2026-02-20T11:59:53.600 INFO:tasks.rados.rados.0.trial153.stdout:160: writing trial15342161-40 from 1768928 to 2174282 tid 2 2026-02-20T11:59:53.605 INFO:tasks.rados.rados.0.trial153.stdout:160: writing trial15342161-40 from 2768204 to 3277714 tid 3 2026-02-20T11:59:53.605 INFO:tasks.rados.rados.0.trial153.stdout:161: read oid 19 snap 11 2026-02-20T11:59:53.605 INFO:tasks.rados.rados.0.trial153.stdout:161: expect (ObjNum 54 snap 0 seq_num 54) 2026-02-20T11:59:53.605 INFO:tasks.rados.rados.0.trial153.stdout:162: write (excl) oid 34 current snap is 13 2026-02-20T11:59:53.605 INFO:tasks.rados.rados.0.trial153.stdout:162: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2026-02-20T11:59:53.608 INFO:tasks.rados.rados.0.trial153.stdout:162: writing trial15342161-34 from 531121 to 994357 tid 1 2026-02-20T11:59:53.613 INFO:tasks.rados.rados.0.trial153.stdout:162: writing trial15342161-34 from 1711057 to 2376422 tid 2 2026-02-20T11:59:53.619 INFO:tasks.rados.rados.0.trial153.stdout:162: writing trial15342161-34 from 3079274 to 3731121 tid 3 2026-02-20T11:59:53.619 INFO:tasks.rados.rados.0.trial153.stdout:163: read oid 5 snap 11 2026-02-20T11:59:53.619 INFO:tasks.rados.rados.0.trial153.stdout:163: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-20T11:59:53.619 INFO:tasks.rados.rados.0.trial153.stdout:164: read oid 16 snap -1 2026-02-20T11:59:53.619 INFO:tasks.rados.rados.0.trial153.stdout:164: expect (ObjNum 89 snap 10 seq_num 89) 2026-02-20T11:59:53.619 INFO:tasks.rados.rados.0.trial153.stdout:165: write oid 4 current snap is 13 2026-02-20T11:59:53.619 INFO:tasks.rados.rados.0.trial153.stdout:158: expect (ObjNum 86 snap 9 seq_num 86) 2026-02-20T11:59:53.619 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:53.666 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 289881 2026-02-20T11:59:53.723 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2018270 2026-02-20T11:59:53.746 INFO:tasks.rados.rados.0.trial153.stdout:165: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2026-02-20T11:59:53.751 INFO:tasks.rados.rados.0.trial153.stdout:165: writing trial15342161-4 from 547928 to 1086413 tid 1 2026-02-20T11:59:53.755 INFO:tasks.rados.rados.0.trial153.stdout:165: writing trial15342161-4 from 1769539 to 2178562 tid 2 2026-02-20T11:59:53.760 INFO:tasks.rados.rados.0.trial153.stdout:165: writing trial15342161-4 from 2706697 to 3238751 tid 3 2026-02-20T11:59:53.760 INFO:tasks.rados.rados.0.trial153.stdout:158: done (6 left) 2026-02-20T11:59:53.760 INFO:tasks.rados.rados.0.trial153.stdout:166: delete oid 10 current snap is 13 2026-02-20T11:59:53.760 INFO:tasks.rados.rados.0.trial153.stdout:163: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-20T11:59:53.771 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1386226 2026-02-20T11:59:53.780 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:53.798 INFO:tasks.rados.rados.0.trial153.stdout:162: finishing write tid 1 to trial15342161-34 2026-02-20T11:59:53.798 INFO:tasks.rados.rados.0.trial153.stdout:160: finishing write tid 1 to trial15342161-40 2026-02-20T11:59:53.798 INFO:tasks.rados.rados.0.trial153.stdout:161: expect (ObjNum 54 snap 0 seq_num 54) 2026-02-20T11:59:53.798 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 246426 2026-02-20T11:59:53.898 INFO:tasks.rados.rados.0.trial153.stdout:162: finishing write tid 2 to trial15342161-34 2026-02-20T11:59:53.898 INFO:tasks.rados.rados.0.trial153.stdout:160: finishing write tid 2 to trial15342161-40 2026-02-20T11:59:53.898 INFO:tasks.rados.rados.0.trial153.stdout:160: finishing write tid 3 to trial15342161-40 2026-02-20T11:59:53.898 INFO:tasks.rados.rados.0.trial153.stdout:162: finishing write tid 3 to trial15342161-34 2026-02-20T11:59:53.898 INFO:tasks.rados.rados.0.trial153.stdout:160: finishing write tid 4 to trial15342161-40 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:160: finishing write tid 5 to trial15342161-40 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:160: oid 40 updating version 0 to 49 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:160: oid 40 version 49 is already newer than 49 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:160: oid 40 version 49 is already newer than 49 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:160: oid 40 version 49 is already newer than 46 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 40 v 49 (ObjNum 102 snap 13 seq_num 102) dirty exists 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:160: left oid 40 (ObjNum 102 snap 13 seq_num 102) 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:162: finishing write tid 4 to trial15342161-34 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:162: finishing write tid 5 to trial15342161-34 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:162: oid 34 updating version 0 to 41 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:162: oid 34 updating version 41 to 44 2026-02-20T11:59:53.899 INFO:tasks.rados.rados.0.trial153.stdout:162: oid 34 version 44 is already newer than 44 2026-02-20T11:59:53.900 INFO:tasks.rados.rados.0.trial153.stdout:162: oid 34 version 44 is already newer than 44 2026-02-20T11:59:53.900 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 44 (ObjNum 103 snap 13 seq_num 103) dirty exists 2026-02-20T11:59:53.900 INFO:tasks.rados.rados.0.trial153.stdout:162: left oid 34 (ObjNum 103 snap 13 seq_num 103) 2026-02-20T11:59:53.900 INFO:tasks.rados.rados.0.trial153.stdout:164: expect (ObjNum 89 snap 10 seq_num 89) 2026-02-20T11:59:54.020 INFO:tasks.rados.rados.0.trial153.stdout:160: done (6 left) 2026-02-20T11:59:54.020 INFO:tasks.rados.rados.0.trial153.stdout:161: done (5 left) 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:162: done (4 left) 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:163: done (3 left) 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:164: done (2 left) 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:166: done (1 left) 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:167: delete oid 28 current snap is 13 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:165: finishing write tid 1 to trial15342161-4 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:165: finishing write tid 2 to trial15342161-4 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:165: finishing write tid 3 to trial15342161-4 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:165: finishing write tid 4 to trial15342161-4 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:165: finishing write tid 5 to trial15342161-4 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:165: oid 4 updating version 0 to 40 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:165: oid 4 version 40 is already newer than 40 2026-02-20T11:59:54.021 INFO:tasks.rados.rados.0.trial153.stdout:165: oid 4 version 40 is already newer than 40 2026-02-20T11:59:54.022 INFO:tasks.rados.rados.0.trial153.stdout:165: oid 4 version 40 is already newer than 40 2026-02-20T11:59:54.022 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 4 v 40 (ObjNum 104 snap 13 seq_num 104) dirty exists 2026-02-20T11:59:54.022 INFO:tasks.rados.rados.0.trial153.stdout:165: left oid 4 (ObjNum 104 snap 13 seq_num 104) 2026-02-20T11:59:54.051 INFO:tasks.rados.rados.0.trial153.stdout:165: done (1 left) 2026-02-20T11:59:54.051 INFO:tasks.rados.rados.0.trial153.stdout:167: done (0 left) 2026-02-20T11:59:54.051 INFO:tasks.rados.rados.0.trial153.stdout:168: write (excl) oid 24 current snap is 13 2026-02-20T11:59:54.052 INFO:tasks.rados.rados.0.trial153.stdout:168: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2026-02-20T11:59:54.055 INFO:tasks.rados.rados.0.trial153.stdout:168: writing trial15342161-24 from 598349 to 1078934 tid 1 2026-02-20T11:59:54.059 INFO:tasks.rados.rados.0.trial153.stdout:168: writing trial15342161-24 from 1544985 to 2217688 tid 2 2026-02-20T11:59:54.064 INFO:tasks.rados.rados.0.trial153.stdout:168: writing trial15342161-24 from 2905319 to 3387542 tid 3 2026-02-20T11:59:54.064 INFO:tasks.rados.rados.0.trial153.stdout:169: snap_remove snap 11 2026-02-20T11:59:54.597 INFO:tasks.rados.rados.0.trial153.stdout:169: done (1 left) 2026-02-20T11:59:54.598 INFO:tasks.rados.rados.0.trial153.stdout:170: write (excl) oid 49 current snap is 13 2026-02-20T11:59:54.598 INFO:tasks.rados.rados.0.trial153.stdout:168: finishing write tid 1 to trial15342161-24 2026-02-20T11:59:54.598 INFO:tasks.rados.rados.0.trial153.stdout:170: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2026-02-20T11:59:54.602 INFO:tasks.rados.rados.0.trial153.stdout:170: writing trial15342161-49 from 615156 to 1170990 tid 1 2026-02-20T11:59:54.605 INFO:tasks.rados.rados.0.trial153.stdout:170: writing trial15342161-49 from 1603467 to 2019828 tid 2 2026-02-20T11:59:54.612 INFO:tasks.rados.rados.0.trial153.stdout:170: writing trial15342161-49 from 2816389 to 3509884 tid 3 2026-02-20T11:59:54.612 INFO:tasks.rados.rados.0.trial153.stdout:171: delete oid 13 current snap is 13 2026-02-20T11:59:54.612 INFO:tasks.rados.rados.0.trial153.stdout:168: finishing write tid 2 to trial15342161-24 2026-02-20T11:59:54.612 INFO:tasks.rados.rados.0.trial153.stdout:168: finishing write tid 3 to trial15342161-24 2026-02-20T11:59:54.612 INFO:tasks.rados.rados.0.trial153.stdout:168: finishing write tid 4 to trial15342161-24 2026-02-20T11:59:54.612 INFO:tasks.rados.rados.0.trial153.stdout:168: finishing write tid 5 to trial15342161-24 2026-02-20T11:59:54.613 INFO:tasks.rados.rados.0.trial153.stdout:168: oid 24 updating version 0 to 9 2026-02-20T11:59:54.613 INFO:tasks.rados.rados.0.trial153.stdout:168: oid 24 version 9 is already newer than 9 2026-02-20T11:59:54.613 INFO:tasks.rados.rados.0.trial153.stdout:168: oid 24 version 9 is already newer than 6 2026-02-20T11:59:54.613 INFO:tasks.rados.rados.0.trial153.stdout:168: oid 24 version 9 is already newer than 9 2026-02-20T11:59:54.613 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 24 v 9 (ObjNum 107 snap 13 seq_num 107) dirty exists 2026-02-20T11:59:54.613 INFO:tasks.rados.rados.0.trial153.stdout:168: left oid 24 (ObjNum 107 snap 13 seq_num 107) 2026-02-20T11:59:54.628 INFO:tasks.rados.rados.0.trial153.stdout:168: done (2 left) 2026-02-20T11:59:54.628 INFO:tasks.rados.rados.0.trial153.stdout:171: done (1 left) 2026-02-20T11:59:54.628 INFO:tasks.rados.rados.0.trial153.stdout:172: delete oid 29 current snap is 13 2026-02-20T11:59:54.640 INFO:tasks.rados.rados.0.trial153.stdout:170: finishing write tid 1 to trial15342161-49 2026-02-20T11:59:54.648 INFO:tasks.rados.rados.0.trial153.stdout:172: done (1 left) 2026-02-20T11:59:54.648 INFO:tasks.rados.rados.0.trial153.stdout:173: read oid 26 snap -1 2026-02-20T11:59:54.648 INFO:tasks.rados.rados.0.trial153.stdout:173: expect (ObjNum 101 snap 11 seq_num 101) 2026-02-20T11:59:54.648 INFO:tasks.rados.rados.0.trial153.stdout:174: write oid 3 current snap is 13 2026-02-20T11:59:54.649 INFO:tasks.rados.rados.0.trial153.stdout:174: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2026-02-20T11:59:54.653 INFO:tasks.rados.rados.0.trial153.stdout:174: writing trial15342161-3 from 665577 to 1447158 tid 1 2026-02-20T11:59:54.658 INFO:tasks.rados.rados.0.trial153.stdout:174: writing trial15342161-3 from 2062560 to 2626248 tid 2 2026-02-20T11:59:54.663 INFO:tasks.rados.rados.0.trial153.stdout:174: writing trial15342161-3 from 3182305 to 3825969 tid 3 2026-02-20T11:59:54.663 INFO:tasks.rados.rados.0.trial153.stdout:175: write (excl) oid 37 current snap is 13 2026-02-20T11:59:54.663 INFO:tasks.rados.rados.0.trial153.stdout:170: finishing write tid 2 to trial15342161-49 2026-02-20T11:59:54.663 INFO:tasks.rados.rados.0.trial153.stdout:175: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2026-02-20T11:59:54.666 INFO:tasks.rados.rados.0.trial153.stdout:175: writing trial15342161-37 from 682384 to 1139214 tid 1 2026-02-20T11:59:54.670 INFO:tasks.rados.rados.0.trial153.stdout:175: writing trial15342161-37 from 1721042 to 2144741 tid 2 2026-02-20T11:59:54.674 INFO:tasks.rados.rados.0.trial153.stdout:175: writing trial15342161-37 from 2809728 to 3264664 tid 3 2026-02-20T11:59:54.677 INFO:tasks.rados.rados.0.trial153.stdout:175: writing trial15342161-37 from 3731357 to 3882384 tid 4 2026-02-20T11:59:54.677 INFO:tasks.rados.rados.0.trial153.stdout:176: delete oid 45 current snap is 13 2026-02-20T11:59:54.678 INFO:tasks.rados.rados.0.trial153.stdout:170: finishing write tid 3 to trial15342161-49 2026-02-20T11:59:54.682 INFO:tasks.rados.rados.0.trial153.stdout:175: finishing write tid 1 to trial15342161-37 2026-02-20T11:59:54.693 INFO:tasks.rados.rados.0.trial153.stdout:170: finishing write tid 4 to trial15342161-49 2026-02-20T11:59:54.696 INFO:tasks.rados.rados.0.trial153.stdout:176: done (4 left) 2026-02-20T11:59:54.696 INFO:tasks.rados.rados.0.trial153.stdout:177: read oid 22 snap -1 2026-02-20T11:59:54.696 INFO:tasks.rados.rados.0.trial153.stdout:177: expect (ObjNum 67 snap 3 seq_num 67) 2026-02-20T11:59:54.696 INFO:tasks.rados.rados.0.trial153.stdout:178: write (excl) oid 13 current snap is 13 2026-02-20T11:59:54.697 INFO:tasks.rados.rados.0.trial153.stdout:178: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2026-02-20T11:59:54.701 INFO:tasks.rados.rados.0.trial153.stdout:178: writing trial15342161-13 from 715998 to 1323326 tid 1 2026-02-20T11:59:54.704 INFO:tasks.rados.rados.0.trial153.stdout:178: writing trial15342161-13 from 1838006 to 2265374 tid 2 2026-02-20T11:59:54.708 INFO:tasks.rados.rados.0.trial153.stdout:178: writing trial15342161-13 from 2864574 to 3342054 tid 3 2026-02-20T11:59:54.711 INFO:tasks.rados.rados.0.trial153.stdout:178: writing trial15342161-13 from 3863835 to 3915998 tid 4 2026-02-20T11:59:54.711 INFO:tasks.rados.rados.0.trial153.stdout:179: write oid 39 current snap is 13 2026-02-20T11:59:54.711 INFO:tasks.rados.rados.0.trial153.stdout:170: finishing write tid 5 to trial15342161-49 2026-02-20T11:59:54.711 INFO:tasks.rados.rados.0.trial153.stdout:170: oid 49 updating version 0 to 30 2026-02-20T11:59:54.711 INFO:tasks.rados.rados.0.trial153.stdout:170: oid 49 version 30 is already newer than 27 2026-02-20T11:59:54.711 INFO:tasks.rados.rados.0.trial153.stdout:170: oid 49 version 30 is already newer than 30 2026-02-20T11:59:54.711 INFO:tasks.rados.rados.0.trial153.stdout:170: oid 49 version 30 is already newer than 30 2026-02-20T11:59:54.711 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 49 v 30 (ObjNum 108 snap 13 seq_num 108) dirty exists 2026-02-20T11:59:54.711 INFO:tasks.rados.rados.0.trial153.stdout:170: left oid 49 (ObjNum 108 snap 13 seq_num 108) 2026-02-20T11:59:54.711 INFO:tasks.rados.rados.0.trial153.stdout:179: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2026-02-20T11:59:54.716 INFO:tasks.rados.rados.0.trial153.stdout:179: writing trial15342161-39 from 732805 to 1531735 tid 1 2026-02-20T11:59:54.721 INFO:tasks.rados.rados.0.trial153.stdout:179: writing trial15342161-39 from 2296488 to 2867514 tid 2 2026-02-20T11:59:54.724 INFO:tasks.rados.rados.0.trial153.stdout:179: writing trial15342161-39 from 3291997 to 3697102 tid 3 2026-02-20T11:59:54.724 INFO:tasks.rados.rados.0.trial153.stdout:170: done (6 left) 2026-02-20T11:59:54.724 INFO:tasks.rados.rados.0.trial153.stdout:180: write (excl) oid 42 current snap is 13 2026-02-20T11:59:54.725 INFO:tasks.rados.rados.0.trial153.stdout:180: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2026-02-20T11:59:54.729 INFO:tasks.rados.rados.0.trial153.stdout:180: writing trial15342161-42 from 749612 to 1223791 tid 1 2026-02-20T11:59:54.733 INFO:tasks.rados.rados.0.trial153.stdout:180: writing trial15342161-42 from 1954970 to 2386007 tid 2 2026-02-20T11:59:54.738 INFO:tasks.rados.rados.0.trial153.stdout:180: writing trial15342161-42 from 2919420 to 3535797 tid 3 2026-02-20T11:59:54.738 INFO:tasks.rados.rados.0.trial153.stdout:181: read oid 32 snap -1 2026-02-20T11:59:54.738 INFO:tasks.rados.rados.0.trial153.stdout:181: expect (ObjNum 76 snap 3 seq_num 76) 2026-02-20T11:59:54.738 INFO:tasks.rados.rados.0.trial153.stdout:182: delete oid 9 current snap is 13 2026-02-20T11:59:54.738 INFO:tasks.rados.rados.0.trial153.stdout:175: finishing write tid 2 to trial15342161-37 2026-02-20T11:59:54.739 INFO:tasks.rados.rados.0.trial153.stdout:175: finishing write tid 3 to trial15342161-37 2026-02-20T11:59:54.739 INFO:tasks.rados.rados.0.trial153.stdout:174: finishing write tid 1 to trial15342161-3 2026-02-20T11:59:54.739 INFO:tasks.rados.rados.0.trial153.stdout:177: expect (ObjNum 67 snap 3 seq_num 67) 2026-02-20T11:59:54.766 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 3071242 2026-02-20T11:59:54.823 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:54.870 INFO:tasks.rados.rados.0.trial153.stdout:173: expect (ObjNum 101 snap 11 seq_num 101) 2026-02-20T11:59:54.870 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:54.921 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:54.997 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2637511 2026-02-20T11:59:55.012 INFO:tasks.rados.rados.0.trial153.stdout:175: finishing write tid 4 to trial15342161-37 2026-02-20T11:59:55.013 INFO:tasks.rados.rados.0.trial153.stdout:173: done (8 left) 2026-02-20T11:59:55.013 INFO:tasks.rados.rados.0.trial153.stdout:177: done (7 left) 2026-02-20T11:59:55.013 INFO:tasks.rados.rados.0.trial153.stdout:182: done (6 left) 2026-02-20T11:59:55.013 INFO:tasks.rados.rados.0.trial153.stdout:183: read oid 17 snap -1 2026-02-20T11:59:55.013 INFO:tasks.rados.rados.0.trial153.stdout:183: expect (ObjNum 16 snap 0 seq_num 16) 2026-02-20T11:59:55.013 INFO:tasks.rados.rados.0.trial153.stdout:184: write (excl) oid 38 current snap is 13 2026-02-20T11:59:55.013 INFO:tasks.rados.rados.0.trial153.stdout:175: finishing write tid 5 to trial15342161-37 2026-02-20T11:59:55.013 INFO:tasks.rados.rados.0.trial153.stdout:184: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2026-02-20T11:59:55.017 INFO:tasks.rados.rados.0.trial153.stdout:184: writing trial15342161-38 from 783226 to 1407903 tid 1 2026-02-20T11:59:55.021 INFO:tasks.rados.rados.0.trial153.stdout:184: writing trial15342161-38 from 2071934 to 2506640 tid 2 2026-02-20T11:59:55.025 INFO:tasks.rados.rados.0.trial153.stdout:184: writing trial15342161-38 from 2974266 to 3613187 tid 3 2026-02-20T11:59:55.025 INFO:tasks.rados.rados.0.trial153.stdout:185: read oid 20 snap -1 2026-02-20T11:59:55.025 INFO:tasks.rados.rados.0.trial153.stdout:185: expect (ObjNum 95 snap 11 seq_num 95) 2026-02-20T11:59:55.025 INFO:tasks.rados.rados.0.trial153.stdout:186: snap_create 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:174: finishing write tid 2 to trial15342161-3 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:175: finishing write tid 6 to trial15342161-37 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:175: oid 37 updating version 0 to 20 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:175: oid 37 version 20 is already newer than 20 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:175: oid 37 version 20 is already newer than 16 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:175: oid 37 version 20 is already newer than 20 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:175: oid 37 version 20 is already newer than 20 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 20 (ObjNum 112 snap 13 seq_num 112) dirty exists 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:175: left oid 37 (ObjNum 112 snap 13 seq_num 112) 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:174: finishing write tid 3 to trial15342161-3 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:178: finishing write tid 1 to trial15342161-13 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:174: finishing write tid 4 to trial15342161-3 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:174: finishing write tid 5 to trial15342161-3 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:174: oid 3 updating version 0 to 44 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:174: oid 3 version 44 is already newer than 44 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:174: oid 3 version 44 is already newer than 44 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:174: oid 3 version 44 is already newer than 44 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 3 v 44 (ObjNum 111 snap 13 seq_num 111) dirty exists 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:174: left oid 3 (ObjNum 111 snap 13 seq_num 111) 2026-02-20T11:59:55.026 INFO:tasks.rados.rados.0.trial153.stdout:181: expect (ObjNum 76 snap 3 seq_num 76) 2026-02-20T11:59:55.053 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:59:55.101 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:55.131 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:55.125+0000 7fc8b04c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:55.134 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:55.129+0000 7fc8b04c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:55.137 INFO:tasks.rados.rados.0.trial153.stdout:178: finishing write tid 2 to trial15342161-13 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:178: finishing write tid 3 to trial15342161-13 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:178: finishing write tid 4 to trial15342161-13 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:178: finishing write tid 5 to trial15342161-13 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:178: finishing write tid 6 to trial15342161-13 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:178: oid 13 updating version 0 to 46 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:178: oid 13 version 46 is already newer than 46 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:178: oid 13 version 46 is already newer than 42 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:178: oid 13 version 46 is already newer than 46 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:178: oid 13 version 46 is already newer than 46 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 13 v 46 (ObjNum 114 snap 13 seq_num 114) dirty exists 2026-02-20T11:59:55.138 INFO:tasks.rados.rados.0.trial153.stdout:178: left oid 13 (ObjNum 114 snap 13 seq_num 114) 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:180: finishing write tid 1 to trial15342161-42 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:179: finishing write tid 1 to trial15342161-39 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:179: finishing write tid 2 to trial15342161-39 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:179: finishing write tid 3 to trial15342161-39 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:179: finishing write tid 4 to trial15342161-39 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:179: finishing write tid 5 to trial15342161-39 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:179: oid 39 updating version 0 to 34 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:179: oid 39 version 34 is already newer than 34 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:179: oid 39 version 34 is already newer than 34 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:179: oid 39 version 34 is already newer than 34 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 39 v 34 (ObjNum 115 snap 13 seq_num 115) dirty exists 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:179: left oid 39 (ObjNum 115 snap 13 seq_num 115) 2026-02-20T11:59:55.139 INFO:tasks.rados.rados.0.trial153.stdout:180: finishing write tid 2 to trial15342161-42 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:180: finishing write tid 3 to trial15342161-42 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:180: finishing write tid 4 to trial15342161-42 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:180: finishing write tid 5 to trial15342161-42 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:180: oid 42 updating version 0 to 33 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:180: oid 42 version 33 is already newer than 30 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:180: oid 42 version 33 is already newer than 33 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:180: oid 42 version 33 is already newer than 33 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 33 (ObjNum 116 snap 13 seq_num 116) dirty exists 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:180: left oid 42 (ObjNum 116 snap 13 seq_num 116) 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:183: expect (ObjNum 16 snap 0 seq_num 16) 2026-02-20T11:59:55.140 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:55.160 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:55.177 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:55.185 INFO:tasks.rados.rados.0.trial153.stdout:184: finishing write tid 1 to trial15342161-38 2026-02-20T11:59:55.185 INFO:tasks.rados.rados.0.trial153.stdout:185: expect (ObjNum 95 snap 11 seq_num 95) 2026-02-20T11:59:55.239 INFO:tasks.rados.rados.0.trial153.stdout:184: finishing write tid 2 to trial15342161-38 2026-02-20T11:59:55.239 INFO:tasks.rados.rados.0.trial153.stdout:184: finishing write tid 3 to trial15342161-38 2026-02-20T11:59:55.239 INFO:tasks.rados.rados.0.trial153.stdout:184: finishing write tid 4 to trial15342161-38 2026-02-20T11:59:55.239 INFO:tasks.rados.rados.0.trial153.stdout:184: finishing write tid 5 to trial15342161-38 2026-02-20T11:59:55.239 INFO:tasks.rados.rados.0.trial153.stdout:184: oid 38 updating version 0 to 48 2026-02-20T11:59:55.240 INFO:tasks.rados.rados.0.trial153.stdout:184: oid 38 version 48 is already newer than 45 2026-02-20T11:59:55.240 INFO:tasks.rados.rados.0.trial153.stdout:184: oid 38 version 48 is already newer than 48 2026-02-20T11:59:55.240 INFO:tasks.rados.rados.0.trial153.stdout:184: oid 38 version 48 is already newer than 48 2026-02-20T11:59:55.240 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 38 v 48 (ObjNum 118 snap 13 seq_num 118) dirty exists 2026-02-20T11:59:55.240 INFO:tasks.rados.rados.0.trial153.stdout:184: left oid 38 (ObjNum 118 snap 13 seq_num 118) 2026-02-20T11:59:55.602 INFO:tasks.rados.rados.0.trial153.stdout:174: done (9 left) 2026-02-20T11:59:55.602 INFO:tasks.rados.rados.0.trial153.stdout:175: done (8 left) 2026-02-20T11:59:55.602 INFO:tasks.rados.rados.0.trial153.stdout:178: done (7 left) 2026-02-20T11:59:55.602 INFO:tasks.rados.rados.0.trial153.stdout:179: done (6 left) 2026-02-20T11:59:55.602 INFO:tasks.rados.rados.0.trial153.stdout:180: done (5 left) 2026-02-20T11:59:55.602 INFO:tasks.rados.rados.0.trial153.stdout:181: done (4 left) 2026-02-20T11:59:55.602 INFO:tasks.rados.rados.0.trial153.stdout:183: done (3 left) 2026-02-20T11:59:55.602 INFO:tasks.rados.rados.0.trial153.stdout:184: done (2 left) 2026-02-20T11:59:55.603 INFO:tasks.rados.rados.0.trial153.stdout:185: done (1 left) 2026-02-20T11:59:55.603 INFO:tasks.rados.rados.0.trial153.stdout:186: done (0 left) 2026-02-20T11:59:55.603 INFO:tasks.rados.rados.0.trial153.stdout:187: read oid 36 snap -1 2026-02-20T11:59:55.603 INFO:tasks.rados.rados.0.trial153.stdout:187: expect (ObjNum 82 snap 8 seq_num 82) 2026-02-20T11:59:55.603 INFO:tasks.rados.rados.0.trial153.stdout:188: read oid 4 snap -1 2026-02-20T11:59:55.603 INFO:tasks.rados.rados.0.trial153.stdout:188: expect (ObjNum 104 snap 13 seq_num 104) 2026-02-20T11:59:55.603 INFO:tasks.rados.rados.0.trial153.stdout:189: read oid 47 snap -1 2026-02-20T11:59:55.603 INFO:tasks.rados.rados.0.trial153.stdout:189: expect (ObjNum 46 snap 0 seq_num 46) 2026-02-20T11:59:55.603 INFO:tasks.rados.rados.0.trial153.stdout:190: read oid 39 snap -1 2026-02-20T11:59:55.603 INFO:tasks.rados.rados.0.trial153.stdout:190: expect (ObjNum 115 snap 13 seq_num 115) 2026-02-20T11:59:55.604 INFO:tasks.rados.rados.0.trial153.stdout:191: read oid 42 snap -1 2026-02-20T11:59:55.604 INFO:tasks.rados.rados.0.trial153.stdout:191: expect (ObjNum 116 snap 13 seq_num 116) 2026-02-20T11:59:55.604 INFO:tasks.rados.rados.0.trial153.stdout:192: snap_create 2026-02-20T11:59:55.627 INFO:tasks.rados.rados.0.trial153.stdout:189: expect (ObjNum 46 snap 0 seq_num 46) 2026-02-20T11:59:55.636 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2534203 2026-02-20T11:59:55.685 INFO:tasks.rados.rados.0.trial153.stdout:190: expect (ObjNum 115 snap 13 seq_num 115) 2026-02-20T11:59:55.719 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:55.777 INFO:tasks.rados.rados.0.trial153.stdout:187: expect (ObjNum 82 snap 8 seq_num 82) 2026-02-20T11:59:55.836 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2876832 2026-02-20T11:59:55.850 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1506613 2026-02-20T11:59:55.876 INFO:tasks.rados.rados.0.trial153.stdout:191: expect (ObjNum 116 snap 13 seq_num 116) 2026-02-20T11:59:55.876 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T11:59:56.067 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 3439587 2026-02-20T11:59:56.075 INFO:tasks.rados.rados.0.trial153.stdout:188: expect (ObjNum 104 snap 13 seq_num 104) 2026-02-20T11:59:56.220 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2357571 2026-02-20T11:59:56.604 INFO:tasks.rados.rados.0.trial153.stdout:187: done (5 left) 2026-02-20T11:59:56.604 INFO:tasks.rados.rados.0.trial153.stdout:188: done (4 left) 2026-02-20T11:59:56.604 INFO:tasks.rados.rados.0.trial153.stdout:189: done (3 left) 2026-02-20T11:59:56.604 INFO:tasks.rados.rados.0.trial153.stdout:190: done (2 left) 2026-02-20T11:59:56.604 INFO:tasks.rados.rados.0.trial153.stdout:191: done (1 left) 2026-02-20T11:59:56.604 INFO:tasks.rados.rados.0.trial153.stdout:192: done (0 left) 2026-02-20T11:59:56.605 INFO:tasks.rados.rados.0.trial153.stdout:193: read oid 44 snap -1 2026-02-20T11:59:56.605 INFO:tasks.rados.rados.0.trial153.stdout:193: expect (ObjNum 66 snap 3 seq_num 66) 2026-02-20T11:59:56.605 INFO:tasks.rados.rados.0.trial153.stdout:194: snap_remove snap 14 2026-02-20T11:59:57.178 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T11:59:57.254 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:57.248+0000 7f5a2a01a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:57.256 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:57.252+0000 7f5a2a01a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:57.282 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:57.298 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:57.608 INFO:tasks.rados.rados.0.trial153.stdout:194: done (1 left) 2026-02-20T11:59:57.608 INFO:tasks.rados.rados.0.trial153.stdout:195: write (excl) oid 17 current snap is 15 2026-02-20T11:59:57.608 INFO:tasks.rados.rados.0.trial153.stdout:195: seq_num 119 ranges {400033=699926,1614063=385970} 2026-02-20T11:59:57.612 INFO:tasks.rados.rados.0.trial153.stdout:195: writing trial15342161-17 from 400033 to 1099959 tid 1 2026-02-20T11:59:57.616 INFO:tasks.rados.rados.0.trial153.stdout:195: writing trial15342161-17 from 1614063 to 2000033 tid 2 2026-02-20T11:59:57.616 INFO:tasks.rados.rados.0.trial153.stdout:196: delete oid 46 current snap is 15 2026-02-20T11:59:57.616 INFO:tasks.rados.rados.0.trial153.stdout:193: expect (ObjNum 66 snap 3 seq_num 66) 2026-02-20T11:59:57.645 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 821450 2026-02-20T11:59:57.683 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1945395 2026-02-20T11:59:57.701 INFO:tasks.rados.rados.0.trial153.stdout:193: done (2 left) 2026-02-20T11:59:57.701 INFO:tasks.rados.rados.0.trial153.stdout:196: done (1 left) 2026-02-20T11:59:57.701 INFO:tasks.rados.rados.0.trial153.stdout:197: snap_remove snap 13 2026-02-20T11:59:57.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~44s 2026-02-20T11:59:57.991 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T11:59:58.063 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T11:59:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T11:59:58.066 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T11:59:58.610 INFO:tasks.rados.rados.0.trial153.stdout:197: done (1 left) 2026-02-20T11:59:58.611 INFO:tasks.rados.rados.0.trial153.stdout:198: delete oid 6 current snap is 15 2026-02-20T11:59:58.611 INFO:tasks.rados.rados.0.trial153.stdout:195: finishing write tid 1 to trial15342161-17 2026-02-20T11:59:58.611 INFO:tasks.rados.rados.0.trial153.stdout:195: finishing write tid 2 to trial15342161-17 2026-02-20T11:59:58.611 INFO:tasks.rados.rados.0.trial153.stdout:195: finishing write tid 3 to trial15342161-17 2026-02-20T11:59:58.611 INFO:tasks.rados.rados.0.trial153.stdout:195: finishing write tid 4 to trial15342161-17 2026-02-20T11:59:58.611 INFO:tasks.rados.rados.0.trial153.stdout:195: oid 17 updating version 0 to 38 2026-02-20T11:59:58.611 INFO:tasks.rados.rados.0.trial153.stdout:195: oid 17 version 38 is already newer than 36 2026-02-20T11:59:58.611 INFO:tasks.rados.rados.0.trial153.stdout:195: oid 17 version 38 is already newer than 38 2026-02-20T11:59:58.612 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 17 v 38 (ObjNum 119 snap 15 seq_num 119) dirty exists 2026-02-20T11:59:58.612 INFO:tasks.rados.rados.0.trial153.stdout:195: left oid 17 (ObjNum 119 snap 15 seq_num 119) 2026-02-20T11:59:58.632 INFO:tasks.rados.rados.0.trial153.stdout:195: done (1 left) 2026-02-20T11:59:58.632 INFO:tasks.rados.rados.0.trial153.stdout:198: done (0 left) 2026-02-20T11:59:58.633 INFO:tasks.rados.rados.0.trial153.stdout:199: write (excl) oid 13 current snap is 15 2026-02-20T11:59:58.633 INFO:tasks.rados.rados.0.trial153.stdout:199: seq_num 122 ranges {450454=642026,1505862=544592} 2026-02-20T11:59:58.636 INFO:tasks.rados.rados.0.trial153.stdout:199: writing trial15342161-13 from 450454 to 1092480 tid 1 2026-02-20T11:59:58.641 INFO:tasks.rados.rados.0.trial153.stdout:199: writing trial15342161-13 from 1505862 to 2050454 tid 2 2026-02-20T11:59:58.641 INFO:tasks.rados.rados.0.trial153.stdout:200: write oid 29 current snap is 15 2026-02-20T11:59:58.641 INFO:tasks.rados.rados.0.trial153.stdout:200: seq_num 123 ranges {467261=717275,1847991=219270} 2026-02-20T11:59:58.645 INFO:tasks.rados.rados.0.trial153.stdout:200: writing trial15342161-29 from 467261 to 1184536 tid 1 2026-02-20T11:59:58.649 INFO:tasks.rados.rados.0.trial153.stdout:200: writing trial15342161-29 from 1847991 to 2067261 tid 2 2026-02-20T11:59:58.649 INFO:tasks.rados.rados.0.trial153.stdout:201: snap_create 2026-02-20T11:59:58.656 INFO:tasks.rados.rados.0.trial153.stdout:199: finishing write tid 1 to trial15342161-13 2026-02-20T11:59:58.670 INFO:tasks.rados.rados.0.trial153.stdout:199: finishing write tid 2 to trial15342161-13 2026-02-20T11:59:58.677 INFO:tasks.rados.rados.0.trial153.stdout:199: finishing write tid 3 to trial15342161-13 2026-02-20T11:59:58.680 INFO:tasks.rados.rados.0.trial153.stdout:199: finishing write tid 4 to trial15342161-13 2026-02-20T11:59:58.680 INFO:tasks.rados.rados.0.trial153.stdout:199: oid 13 updating version 0 to 50 2026-02-20T11:59:58.680 INFO:tasks.rados.rados.0.trial153.stdout:199: oid 13 version 50 is already newer than 48 2026-02-20T11:59:58.681 INFO:tasks.rados.rados.0.trial153.stdout:199: oid 13 version 50 is already newer than 50 2026-02-20T11:59:58.681 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 13 v 50 (ObjNum 122 snap 15 seq_num 122) dirty exists 2026-02-20T11:59:58.681 INFO:tasks.rados.rados.0.trial153.stdout:199: left oid 13 (ObjNum 122 snap 15 seq_num 122) 2026-02-20T11:59:58.683 INFO:tasks.rados.rados.0.trial153.stdout:200: finishing write tid 1 to trial15342161-29 2026-02-20T11:59:58.688 INFO:tasks.rados.rados.0.trial153.stdout:200: finishing write tid 2 to trial15342161-29 2026-02-20T11:59:58.692 INFO:tasks.rados.rados.0.trial153.stdout:200: finishing write tid 3 to trial15342161-29 2026-02-20T11:59:58.693 INFO:tasks.rados.rados.0.trial153.stdout:200: finishing write tid 4 to trial15342161-29 2026-02-20T11:59:58.693 INFO:tasks.rados.rados.0.trial153.stdout:200: oid 29 updating version 0 to 53 2026-02-20T11:59:58.694 INFO:tasks.rados.rados.0.trial153.stdout:200: oid 29 version 53 is already newer than 53 2026-02-20T11:59:58.694 INFO:tasks.rados.rados.0.trial153.stdout:200: oid 29 version 53 is already newer than 53 2026-02-20T11:59:58.694 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 29 v 53 (ObjNum 123 snap 15 seq_num 123) dirty exists 2026-02-20T11:59:58.694 INFO:tasks.rados.rados.0.trial153.stdout:200: left oid 29 (ObjNum 123 snap 15 seq_num 123) 2026-02-20T11:59:59.299 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T11:59:59.377 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:59.371+0000 7fd78d7dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:59.379 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T11:59:59.375+0000 7fd78d7dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T11:59:59.407 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T11:59:59.423 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T11:59:59.423 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T11:59:59.616 INFO:tasks.rados.rados.0.trial153.stdout:199: done (2 left) 2026-02-20T11:59:59.616 INFO:tasks.rados.rados.0.trial153.stdout:200: done (1 left) 2026-02-20T11:59:59.616 INFO:tasks.rados.rados.0.trial153.stdout:201: done (0 left) 2026-02-20T11:59:59.616 INFO:tasks.rados.rados.0.trial153.stdout:202: write (excl) oid 45 current snap is 16 2026-02-20T11:59:59.616 INFO:tasks.rados.rados.0.trial153.stdout:202: seq_num 124 ranges {484068=792524,1906473=177595} 2026-02-20T11:59:59.620 INFO:tasks.rados.rados.0.trial153.stdout:202: writing trial15342161-45 from 484068 to 1276592 tid 1 2026-02-20T11:59:59.624 INFO:tasks.rados.rados.0.trial153.stdout:202: writing trial15342161-45 from 1906473 to 2084068 tid 2 2026-02-20T11:59:59.624 INFO:tasks.rados.rados.0.trial153.stdout:203: snap_remove snap 15 2026-02-20T12:00:00.617 INFO:tasks.rados.rados.0.trial153.stdout:203: done (1 left) 2026-02-20T12:00:00.617 INFO:tasks.rados.rados.0.trial153.stdout:204: write (excl) oid 9 current snap is 16 2026-02-20T12:00:00.617 INFO:tasks.rados.rados.0.trial153.stdout:202: finishing write tid 1 to trial15342161-45 2026-02-20T12:00:00.617 INFO:tasks.rados.rados.0.trial153.stdout:204: seq_num 125 ranges {500875=467773,1564955=535920} 2026-02-20T12:00:00.621 INFO:tasks.rados.rados.0.trial153.stdout:204: writing trial15342161-9 from 500875 to 968648 tid 1 2026-02-20T12:00:00.625 INFO:tasks.rados.rados.0.trial153.stdout:204: writing trial15342161-9 from 1564955 to 2100875 tid 2 2026-02-20T12:00:00.626 INFO:tasks.rados.rados.0.trial153.stdout:205: write (excl) oid 20 current snap is 16 2026-02-20T12:00:00.626 INFO:tasks.rados.rados.0.trial153.stdout:202: finishing write tid 2 to trial15342161-45 2026-02-20T12:00:00.626 INFO:tasks.rados.rados.0.trial153.stdout:205: seq_num 126 ranges {517682=543022,1623437=494245} 2026-02-20T12:00:00.629 INFO:tasks.rados.rados.0.trial153.stdout:205: writing trial15342161-20 from 517682 to 1060704 tid 1 2026-02-20T12:00:00.633 INFO:tasks.rados.rados.0.trial153.stdout:205: writing trial15342161-20 from 1623437 to 2117682 tid 2 2026-02-20T12:00:00.633 INFO:tasks.rados.rados.0.trial153.stdout:206: snap_create 2026-02-20T12:00:00.634 INFO:tasks.rados.rados.0.trial153.stdout:202: finishing write tid 3 to trial15342161-45 2026-02-20T12:00:00.634 INFO:tasks.rados.rados.0.trial153.stdout:202: finishing write tid 4 to trial15342161-45 2026-02-20T12:00:00.634 INFO:tasks.rados.rados.0.trial153.stdout:202: oid 45 updating version 0 to 23 2026-02-20T12:00:00.634 INFO:tasks.rados.rados.0.trial153.stdout:202: oid 45 version 23 is already newer than 21 2026-02-20T12:00:00.634 INFO:tasks.rados.rados.0.trial153.stdout:202: oid 45 version 23 is already newer than 23 2026-02-20T12:00:00.634 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 23 (ObjNum 124 snap 16 seq_num 124) dirty exists 2026-02-20T12:00:00.634 INFO:tasks.rados.rados.0.trial153.stdout:202: left oid 45 (ObjNum 124 snap 16 seq_num 124) 2026-02-20T12:00:00.644 INFO:tasks.rados.rados.0.trial153.stdout:205: finishing write tid 1 to trial15342161-20 2026-02-20T12:00:00.644 INFO:tasks.rados.rados.0.trial153.stdout:204: finishing write tid 1 to trial15342161-9 2026-02-20T12:00:00.658 INFO:tasks.rados.rados.0.trial153.stdout:204: finishing write tid 2 to trial15342161-9 2026-02-20T12:00:00.658 INFO:tasks.rados.rados.0.trial153.stdout:205: finishing write tid 2 to trial15342161-20 2026-02-20T12:00:00.662 INFO:tasks.rados.rados.0.trial153.stdout:204: finishing write tid 3 to trial15342161-9 2026-02-20T12:00:00.663 INFO:tasks.rados.rados.0.trial153.stdout:204: finishing write tid 4 to trial15342161-9 2026-02-20T12:00:00.663 INFO:tasks.rados.rados.0.trial153.stdout:204: oid 9 updating version 0 to 56 2026-02-20T12:00:00.663 INFO:tasks.rados.rados.0.trial153.stdout:204: oid 9 version 56 is already newer than 56 2026-02-20T12:00:00.663 INFO:tasks.rados.rados.0.trial153.stdout:204: oid 9 version 56 is already newer than 54 2026-02-20T12:00:00.663 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 9 v 56 (ObjNum 125 snap 16 seq_num 125) dirty exists 2026-02-20T12:00:00.663 INFO:tasks.rados.rados.0.trial153.stdout:204: left oid 9 (ObjNum 125 snap 16 seq_num 125) 2026-02-20T12:00:00.671 INFO:tasks.rados.rados.0.trial153.stdout:205: finishing write tid 3 to trial15342161-20 2026-02-20T12:00:00.673 INFO:tasks.rados.rados.0.trial153.stdout:205: finishing write tid 4 to trial15342161-20 2026-02-20T12:00:00.673 INFO:tasks.rados.rados.0.trial153.stdout:205: oid 20 updating version 0 to 21 2026-02-20T12:00:00.673 INFO:tasks.rados.rados.0.trial153.stdout:205: oid 20 version 21 is already newer than 21 2026-02-20T12:00:00.673 INFO:tasks.rados.rados.0.trial153.stdout:205: oid 20 version 21 is already newer than 19 2026-02-20T12:00:00.673 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 20 v 21 (ObjNum 126 snap 16 seq_num 126) dirty exists 2026-02-20T12:00:00.673 INFO:tasks.rados.rados.0.trial153.stdout:205: left oid 20 (ObjNum 126 snap 16 seq_num 126) 2026-02-20T12:00:01.423 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:01.503 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:01.497+0000 7fc18457e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:01.505 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:01.501+0000 7fc18457e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:01.531 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:01.547 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:01.621 INFO:tasks.rados.rados.0.trial153.stdout:202: done (3 left) 2026-02-20T12:00:01.621 INFO:tasks.rados.rados.0.trial153.stdout:204: done (2 left) 2026-02-20T12:00:01.622 INFO:tasks.rados.rados.0.trial153.stdout:205: done (1 left) 2026-02-20T12:00:01.622 INFO:tasks.rados.rados.0.trial153.stdout:206: done (0 left) 2026-02-20T12:00:01.622 INFO:tasks.rados.rados.0.trial153.stdout:207: snap_create 2026-02-20T12:00:02.624 INFO:tasks.rados.rados.0.trial153.stdout:207: done (0 left) 2026-02-20T12:00:02.624 INFO:tasks.rados.rados.0.trial153.stdout:208: snap_remove snap 17 2026-02-20T12:00:03.067 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:03.142 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:03.145 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:03.146 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:03.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2026-02-20T12:00:03.547 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:03.627 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:03.621+0000 7f440685a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:03.627 INFO:tasks.rados.rados.0.trial153.stdout:208: done (0 left) 2026-02-20T12:00:03.627 INFO:tasks.rados.rados.0.trial153.stdout:209: write (excl) oid 9 current snap is 18 2026-02-20T12:00:03.627 INFO:tasks.rados.rados.0.trial153.stdout:209: seq_num 127 ranges {534489=618271,1565566=568923} 2026-02-20T12:00:03.630 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:03.625+0000 7f440685a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:03.632 INFO:tasks.rados.rados.0.trial153.stdout:209: writing trial15342161-9 from 534489 to 1152760 tid 1 2026-02-20T12:00:03.636 INFO:tasks.rados.rados.0.trial153.stdout:209: writing trial15342161-9 from 1565566 to 2134489 tid 2 2026-02-20T12:00:03.636 INFO:tasks.rados.rados.0.trial153.stdout:210: snap_create 2026-02-20T12:00:03.676 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:03.679 INFO:tasks.rados.rados.0.trial153.stdout:209: finishing write tid 1 to trial15342161-9 2026-02-20T12:00:03.695 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:03.696 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:03.703 INFO:tasks.rados.rados.0.trial153.stdout:209: finishing write tid 2 to trial15342161-9 2026-02-20T12:00:03.707 INFO:tasks.rados.rados.0.trial153.stdout:209: finishing write tid 3 to trial15342161-9 2026-02-20T12:00:03.708 INFO:tasks.rados.rados.0.trial153.stdout:209: finishing write tid 4 to trial15342161-9 2026-02-20T12:00:03.708 INFO:tasks.rados.rados.0.trial153.stdout:209: oid 9 updating version 0 to 60 2026-02-20T12:00:03.708 INFO:tasks.rados.rados.0.trial153.stdout:209: oid 9 version 60 is already newer than 58 2026-02-20T12:00:03.708 INFO:tasks.rados.rados.0.trial153.stdout:209: oid 9 version 60 is already newer than 60 2026-02-20T12:00:03.708 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 9 v 60 (ObjNum 127 snap 18 seq_num 127) dirty exists 2026-02-20T12:00:03.708 INFO:tasks.rados.rados.0.trial153.stdout:209: left oid 9 (ObjNum 127 snap 18 seq_num 127) 2026-02-20T12:00:04.630 INFO:tasks.rados.rados.0.trial153.stdout:209: done (1 left) 2026-02-20T12:00:04.631 INFO:tasks.rados.rados.0.trial153.stdout:210: done (0 left) 2026-02-20T12:00:04.631 INFO:tasks.rados.rados.0.trial153.stdout:211: read oid 33 snap -1 2026-02-20T12:00:04.631 INFO:tasks.rados.rados.0.trial153.stdout:211: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T12:00:04.631 INFO:tasks.rados.rados.0.trial153.stdout:212: snap_remove snap 16 2026-02-20T12:00:05.636 INFO:tasks.rados.rados.0.trial153.stdout:212: done (1 left) 2026-02-20T12:00:05.636 INFO:tasks.rados.rados.0.trial153.stdout:213: write (excl) oid 31 current snap is 19 2026-02-20T12:00:05.636 INFO:tasks.rados.rados.0.trial153.stdout:213: seq_num 128 ranges {551296=693520,2024048=127248} 2026-02-20T12:00:05.641 INFO:tasks.rados.rados.0.trial153.stdout:213: writing trial15342161-31 from 551296 to 1244816 tid 1 2026-02-20T12:00:05.644 INFO:tasks.rados.rados.0.trial153.stdout:213: writing trial15342161-31 from 2024048 to 2151296 tid 2 2026-02-20T12:00:05.644 INFO:tasks.rados.rados.0.trial153.stdout:214: delete oid 28 current snap is 19 2026-02-20T12:00:05.644 INFO:tasks.rados.rados.0.trial153.stdout:211: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T12:00:05.667 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:05.696 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:05.711 INFO:tasks.rados.rados.0.trial153.stdout:213: finishing write tid 1 to trial15342161-31 2026-02-20T12:00:05.711 INFO:tasks.rados.rados.0.trial153.stdout:213: finishing write tid 2 to trial15342161-31 2026-02-20T12:00:05.711 INFO:tasks.rados.rados.0.trial153.stdout:213: finishing write tid 3 to trial15342161-31 2026-02-20T12:00:05.711 INFO:tasks.rados.rados.0.trial153.stdout:211: done (2 left) 2026-02-20T12:00:05.711 INFO:tasks.rados.rados.0.trial153.stdout:214: done (1 left) 2026-02-20T12:00:05.711 INFO:tasks.rados.rados.0.trial153.stdout:215: delete oid 42 current snap is 19 2026-02-20T12:00:05.711 INFO:tasks.rados.rados.0.trial153.stdout:213: finishing write tid 4 to trial15342161-31 2026-02-20T12:00:05.712 INFO:tasks.rados.rados.0.trial153.stdout:213: oid 31 updating version 0 to 25 2026-02-20T12:00:05.712 INFO:tasks.rados.rados.0.trial153.stdout:213: oid 31 version 25 is already newer than 25 2026-02-20T12:00:05.712 INFO:tasks.rados.rados.0.trial153.stdout:213: oid 31 version 25 is already newer than 23 2026-02-20T12:00:05.712 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 31 v 25 (ObjNum 128 snap 19 seq_num 128) dirty exists 2026-02-20T12:00:05.712 INFO:tasks.rados.rados.0.trial153.stdout:213: left oid 31 (ObjNum 128 snap 19 seq_num 128) 2026-02-20T12:00:05.741 INFO:tasks.rados.rados.0.trial153.stdout:213: done (1 left) 2026-02-20T12:00:05.741 INFO:tasks.rados.rados.0.trial153.stdout:215: done (0 left) 2026-02-20T12:00:05.741 INFO:tasks.rados.rados.0.trial153.stdout:216: read oid 3 snap -1 2026-02-20T12:00:05.741 INFO:tasks.rados.rados.0.trial153.stdout:216: expect (ObjNum 111 snap 13 seq_num 111) 2026-02-20T12:00:05.741 INFO:tasks.rados.rados.0.trial153.stdout:217: write (excl) oid 37 current snap is 19 2026-02-20T12:00:05.741 INFO:tasks.rados.rados.0.trial153.stdout:217: seq_num 131 ranges {601717=635620,1799494=402223} 2026-02-20T12:00:05.745 INFO:tasks.rados.rados.0.trial153.stdout:217: writing trial15342161-37 from 601717 to 1237337 tid 1 2026-02-20T12:00:05.749 INFO:tasks.rados.rados.0.trial153.stdout:217: writing trial15342161-37 from 1799494 to 2201717 tid 2 2026-02-20T12:00:05.749 INFO:tasks.rados.rados.0.trial153.stdout:218: write (excl) oid 11 current snap is 19 2026-02-20T12:00:05.749 INFO:tasks.rados.rados.0.trial153.stdout:218: seq_num 132 ranges {618524=710869,1857976=360548} 2026-02-20T12:00:05.754 INFO:tasks.rados.rados.0.trial153.stdout:218: writing trial15342161-11 from 618524 to 1329393 tid 1 2026-02-20T12:00:05.757 INFO:tasks.rados.rados.0.trial153.stdout:218: writing trial15342161-11 from 1857976 to 2218524 tid 2 2026-02-20T12:00:05.758 INFO:tasks.rados.rados.0.trial153.stdout:219: snap_remove snap 18 2026-02-20T12:00:05.777 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:05.770+0000 7f37d07b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:05.779 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:05.774+0000 7f37d07b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:05.821 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:05.835 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:05.836 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:06.638 INFO:tasks.rados.rados.0.trial153.stdout:219: done (3 left) 2026-02-20T12:00:06.638 INFO:tasks.rados.rados.0.trial153.stdout:220: write (excl) oid 19 current snap is 19 2026-02-20T12:00:06.639 INFO:tasks.rados.rados.0.trial153.stdout:220: seq_num 133 ranges {635331=786118,1916458=318873} 2026-02-20T12:00:06.643 INFO:tasks.rados.rados.0.trial153.stdout:220: writing trial15342161-19 from 635331 to 1421449 tid 1 2026-02-20T12:00:06.646 INFO:tasks.rados.rados.0.trial153.stdout:220: writing trial15342161-19 from 1916458 to 2235331 tid 2 2026-02-20T12:00:06.646 INFO:tasks.rados.rados.0.trial153.stdout:221: write oid 23 current snap is 19 2026-02-20T12:00:06.646 INFO:tasks.rados.rados.0.trial153.stdout:216: expect (ObjNum 111 snap 13 seq_num 111) 2026-02-20T12:00:06.653 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1131047 2026-02-20T12:00:06.751 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 3169909 2026-02-20T12:00:06.762 INFO:tasks.rados.rados.0.trial153.stdout:217: finishing write tid 1 to trial15342161-37 2026-02-20T12:00:06.763 INFO:tasks.rados.rados.0.trial153.stdout:221: seq_num 134 ranges {652138=461367,1574940=677198} 2026-02-20T12:00:06.767 INFO:tasks.rados.rados.0.trial153.stdout:221: writing trial15342161-23 from 652138 to 1113505 tid 1 2026-02-20T12:00:06.772 INFO:tasks.rados.rados.0.trial153.stdout:221: writing trial15342161-23 from 1574940 to 2252138 tid 2 2026-02-20T12:00:06.772 INFO:tasks.rados.rados.0.trial153.stdout:216: done (4 left) 2026-02-20T12:00:06.772 INFO:tasks.rados.rados.0.trial153.stdout:222: snap_create 2026-02-20T12:00:06.772 INFO:tasks.rados.rados.0.trial153.stdout:218: finishing write tid 1 to trial15342161-11 2026-02-20T12:00:06.772 INFO:tasks.rados.rados.0.trial153.stdout:217: finishing write tid 2 to trial15342161-37 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:217: finishing write tid 3 to trial15342161-37 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:218: finishing write tid 2 to trial15342161-11 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:217: finishing write tid 4 to trial15342161-37 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:217: oid 37 updating version 0 to 22 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:217: oid 37 updating version 22 to 24 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:217: oid 37 version 24 is already newer than 24 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 24 (ObjNum 131 snap 19 seq_num 131) dirty exists 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:217: left oid 37 (ObjNum 131 snap 19 seq_num 131) 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:218: finishing write tid 3 to trial15342161-11 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:218: finishing write tid 4 to trial15342161-11 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:218: oid 11 updating version 0 to 16 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:218: oid 11 version 16 is already newer than 16 2026-02-20T12:00:06.773 INFO:tasks.rados.rados.0.trial153.stdout:218: oid 11 version 16 is already newer than 14 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 11 v 16 (ObjNum 132 snap 19 seq_num 132) dirty exists 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:218: left oid 11 (ObjNum 132 snap 19 seq_num 132) 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:220: finishing write tid 1 to trial15342161-19 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:220: finishing write tid 2 to trial15342161-19 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:220: finishing write tid 3 to trial15342161-19 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:220: finishing write tid 4 to trial15342161-19 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:220: oid 19 updating version 0 to 36 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:220: oid 19 version 36 is already newer than 36 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:220: oid 19 version 36 is already newer than 34 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 36 (ObjNum 133 snap 19 seq_num 133) dirty exists 2026-02-20T12:00:06.774 INFO:tasks.rados.rados.0.trial153.stdout:220: left oid 19 (ObjNum 133 snap 19 seq_num 133) 2026-02-20T12:00:06.786 INFO:tasks.rados.rados.0.trial153.stdout:221: finishing write tid 1 to trial15342161-23 2026-02-20T12:00:06.793 INFO:tasks.rados.rados.0.trial153.stdout:221: finishing write tid 2 to trial15342161-23 2026-02-20T12:00:06.799 INFO:tasks.rados.rados.0.trial153.stdout:221: finishing write tid 3 to trial15342161-23 2026-02-20T12:00:06.800 INFO:tasks.rados.rados.0.trial153.stdout:221: finishing write tid 4 to trial15342161-23 2026-02-20T12:00:06.800 INFO:tasks.rados.rados.0.trial153.stdout:221: oid 23 updating version 0 to 54 2026-02-20T12:00:06.800 INFO:tasks.rados.rados.0.trial153.stdout:221: oid 23 version 54 is already newer than 54 2026-02-20T12:00:06.800 INFO:tasks.rados.rados.0.trial153.stdout:221: oid 23 version 54 is already newer than 54 2026-02-20T12:00:06.801 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 23 v 54 (ObjNum 134 snap 19 seq_num 134) dirty exists 2026-02-20T12:00:06.801 INFO:tasks.rados.rados.0.trial153.stdout:221: left oid 23 (ObjNum 134 snap 19 seq_num 134) 2026-02-20T12:00:07.641 INFO:tasks.rados.rados.0.trial153.stdout:217: done (4 left) 2026-02-20T12:00:07.641 INFO:tasks.rados.rados.0.trial153.stdout:218: done (3 left) 2026-02-20T12:00:07.641 INFO:tasks.rados.rados.0.trial153.stdout:220: done (2 left) 2026-02-20T12:00:07.641 INFO:tasks.rados.rados.0.trial153.stdout:221: done (1 left) 2026-02-20T12:00:07.642 INFO:tasks.rados.rados.0.trial153.stdout:222: done (0 left) 2026-02-20T12:00:07.642 INFO:tasks.rados.rados.0.trial153.stdout:223: read oid 13 snap -1 2026-02-20T12:00:07.642 INFO:tasks.rados.rados.0.trial153.stdout:223: expect (ObjNum 122 snap 15 seq_num 122) 2026-02-20T12:00:07.642 INFO:tasks.rados.rados.0.trial153.stdout:224: delete oid 37 current snap is 20 2026-02-20T12:00:07.679 INFO:tasks.rados.rados.0.trial153.stdout:223: expect (ObjNum 122 snap 15 seq_num 122) 2026-02-20T12:00:07.712 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1955736 2026-02-20T12:00:07.735 INFO:tasks.rados.rados.0.trial153.stdout:223: done (1 left) 2026-02-20T12:00:07.736 INFO:tasks.rados.rados.0.trial153.stdout:224: done (0 left) 2026-02-20T12:00:07.736 INFO:tasks.rados.rados.0.trial153.stdout:225: read oid 31 snap 19 2026-02-20T12:00:07.736 INFO:tasks.rados.rados.0.trial153.stdout:225: expect (ObjNum 128 snap 19 seq_num 128) 2026-02-20T12:00:07.736 INFO:tasks.rados.rados.0.trial153.stdout:226: read oid 10 snap -1 2026-02-20T12:00:07.736 INFO:tasks.rados.rados.0.trial153.stdout:226: expect deleted 2026-02-20T12:00:07.736 INFO:tasks.rados.rados.0.trial153.stdout:227: write oid 45 current snap is 20 2026-02-20T12:00:07.737 INFO:tasks.rados.rados.0.trial153.stdout:227: seq_num 136 ranges {685752=611865,1975551=310201} 2026-02-20T12:00:07.740 INFO:tasks.rados.rados.0.trial153.stdout:227: writing trial15342161-45 from 685752 to 1297617 tid 1 2026-02-20T12:00:07.744 INFO:tasks.rados.rados.0.trial153.stdout:227: writing trial15342161-45 from 1975551 to 2285752 tid 2 2026-02-20T12:00:07.745 INFO:tasks.rados.rados.0.trial153.stdout:228: write (excl) oid 1 current snap is 20 2026-02-20T12:00:07.745 INFO:tasks.rados.rados.0.trial153.stdout:228: seq_num 137 ranges {702559=403467,1750386=495032} 2026-02-20T12:00:07.748 INFO:tasks.rados.rados.0.trial153.stdout:228: writing trial15342161-1 from 702559 to 1106026 tid 1 2026-02-20T12:00:07.753 INFO:tasks.rados.rados.0.trial153.stdout:228: writing trial15342161-1 from 1750386 to 2245418 tid 2 2026-02-20T12:00:07.753 INFO:tasks.rados.rados.0.trial153.stdout:229: write (excl) oid 38 current snap is 20 2026-02-20T12:00:07.753 INFO:tasks.rados.rados.0.trial153.stdout:229: seq_num 138 ranges {719366=478716,1808868=510498} 2026-02-20T12:00:07.757 INFO:tasks.rados.rados.0.trial153.stdout:229: writing trial15342161-38 from 719366 to 1198082 tid 1 2026-02-20T12:00:07.761 INFO:tasks.rados.rados.0.trial153.stdout:229: writing trial15342161-38 from 1808868 to 2319366 tid 2 2026-02-20T12:00:07.761 INFO:tasks.rados.rados.0.trial153.stdout:230: snap_create 2026-02-20T12:00:07.761 INFO:tasks.rados.rados.0.trial153.stdout:225: expect (ObjNum 128 snap 19 seq_num 128) 2026-02-20T12:00:07.761 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:07.795 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1929009 2026-02-20T12:00:07.823 INFO:tasks.rados.rados.0.trial153.stdout:227: finishing write tid 1 to trial15342161-45 2026-02-20T12:00:07.823 INFO:tasks.rados.rados.0.trial153.stdout:228: finishing write tid 1 to trial15342161-1 2026-02-20T12:00:07.823 INFO:tasks.rados.rados.0.trial153.stdout:227: finishing write tid 2 to trial15342161-45 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:229: finishing write tid 1 to trial15342161-38 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:227: finishing write tid 3 to trial15342161-45 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:228: finishing write tid 2 to trial15342161-1 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:227: finishing write tid 4 to trial15342161-45 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:227: oid 45 updating version 0 to 27 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:227: oid 45 version 27 is already newer than 27 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:227: oid 45 version 27 is already newer than 27 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 27 (ObjNum 136 snap 20 seq_num 136) dirty exists 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:227: left oid 45 (ObjNum 136 snap 20 seq_num 136) 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:228: finishing write tid 3 to trial15342161-1 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:228: finishing write tid 4 to trial15342161-1 2026-02-20T12:00:07.824 INFO:tasks.rados.rados.0.trial153.stdout:228: oid 1 updating version 0 to 66 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:228: oid 1 version 66 is already newer than 66 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:228: oid 1 version 66 is already newer than 64 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 1 v 66 (ObjNum 137 snap 20 seq_num 137) dirty exists 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:228: left oid 1 (ObjNum 137 snap 20 seq_num 137) 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:229: finishing write tid 2 to trial15342161-38 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:229: finishing write tid 3 to trial15342161-38 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:229: finishing write tid 4 to trial15342161-38 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:229: oid 38 updating version 0 to 52 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:229: oid 38 version 52 is already newer than 52 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:229: oid 38 version 52 is already newer than 50 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 38 v 52 (ObjNum 138 snap 20 seq_num 138) dirty exists 2026-02-20T12:00:07.825 INFO:tasks.rados.rados.0.trial153.stdout:229: left oid 38 (ObjNum 138 snap 20 seq_num 138) 2026-02-20T12:00:07.837 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:07.915 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:07.909+0000 7f59ac6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:07.918 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:07.913+0000 7f59ac6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:07.945 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:07.962 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:08.147 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:08.220 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:08.223 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:08.643 INFO:tasks.rados.rados.0.trial153.stdout:225: done (5 left) 2026-02-20T12:00:08.644 INFO:tasks.rados.rados.0.trial153.stdout:226: done (4 left) 2026-02-20T12:00:08.644 INFO:tasks.rados.rados.0.trial153.stdout:227: done (3 left) 2026-02-20T12:00:08.644 INFO:tasks.rados.rados.0.trial153.stdout:228: done (2 left) 2026-02-20T12:00:08.644 INFO:tasks.rados.rados.0.trial153.stdout:229: done (1 left) 2026-02-20T12:00:08.644 INFO:tasks.rados.rados.0.trial153.stdout:230: done (0 left) 2026-02-20T12:00:08.644 INFO:tasks.rados.rados.0.trial153.stdout:231: read oid 33 snap -1 2026-02-20T12:00:08.644 INFO:tasks.rados.rados.0.trial153.stdout:231: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T12:00:08.644 INFO:tasks.rados.rados.0.trial153.stdout:232: delete oid 2 current snap is 21 2026-02-20T12:00:08.672 INFO:tasks.rados.rados.0.trial153.stdout:231: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-20T12:00:08.680 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2191644 2026-02-20T12:00:08.683 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:08.725 INFO:tasks.rados.rados.0.trial153.stdout:231: done (1 left) 2026-02-20T12:00:08.725 INFO:tasks.rados.rados.0.trial153.stdout:232: done (0 left) 2026-02-20T12:00:08.725 INFO:tasks.rados.rados.0.trial153.stdout:233: write (excl) oid 35 current snap is 21 2026-02-20T12:00:08.725 INFO:tasks.rados.rados.0.trial153.stdout:233: seq_num 140 ranges {752980=629214,1809479=543501} 2026-02-20T12:00:08.730 INFO:tasks.rados.rados.0.trial153.stdout:233: writing trial15342161-35 from 752980 to 1382194 tid 1 2026-02-20T12:00:08.734 INFO:tasks.rados.rados.0.trial153.stdout:233: writing trial15342161-35 from 1809479 to 2352980 tid 2 2026-02-20T12:00:08.734 INFO:tasks.rados.rados.0.trial153.stdout:234: snap_remove snap 20 2026-02-20T12:00:08.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~55s 2026-02-20T12:00:09.646 INFO:tasks.rados.rados.0.trial153.stdout:234: done (1 left) 2026-02-20T12:00:09.646 INFO:tasks.rados.rados.0.trial153.stdout:235: read oid 6 snap -1 2026-02-20T12:00:09.646 INFO:tasks.rados.rados.0.trial153.stdout:235: expect deleted 2026-02-20T12:00:09.646 INFO:tasks.rados.rados.0.trial153.stdout:233: finishing write tid 1 to trial15342161-35 2026-02-20T12:00:09.646 INFO:tasks.rados.rados.0.trial153.stdout:236: read oid 28 snap -1 2026-02-20T12:00:09.646 INFO:tasks.rados.rados.0.trial153.stdout:236: expect deleted 2026-02-20T12:00:09.646 INFO:tasks.rados.rados.0.trial153.stdout:237: write oid 25 current snap is 21 2026-02-20T12:00:09.646 INFO:tasks.rados.rados.0.trial153.stdout:233: finishing write tid 2 to trial15342161-35 2026-02-20T12:00:09.646 INFO:tasks.rados.rados.0.trial153.stdout:237: seq_num 141 ranges {769787=704463,2267961=101826} 2026-02-20T12:00:09.652 INFO:tasks.rados.rados.0.trial153.stdout:237: writing trial15342161-25 from 769787 to 1474250 tid 1 2026-02-20T12:00:09.655 INFO:tasks.rados.rados.0.trial153.stdout:237: writing trial15342161-25 from 2267961 to 2369787 tid 2 2026-02-20T12:00:09.656 INFO:tasks.rados.rados.0.trial153.stdout:238: read oid 29 snap -1 2026-02-20T12:00:09.656 INFO:tasks.rados.rados.0.trial153.stdout:238: expect (ObjNum 123 snap 15 seq_num 123) 2026-02-20T12:00:09.656 INFO:tasks.rados.rados.0.trial153.stdout:239: delete oid 30 current snap is 21 2026-02-20T12:00:09.656 INFO:tasks.rados.rados.0.trial153.stdout:233: finishing write tid 3 to trial15342161-35 2026-02-20T12:00:09.656 INFO:tasks.rados.rados.0.trial153.stdout:233: finishing write tid 4 to trial15342161-35 2026-02-20T12:00:09.656 INFO:tasks.rados.rados.0.trial153.stdout:233: oid 35 updating version 0 to 56 2026-02-20T12:00:09.656 INFO:tasks.rados.rados.0.trial153.stdout:233: oid 35 updating version 56 to 58 2026-02-20T12:00:09.656 INFO:tasks.rados.rados.0.trial153.stdout:233: oid 35 version 58 is already newer than 58 2026-02-20T12:00:09.656 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 35 v 58 (ObjNum 140 snap 21 seq_num 140) dirty exists 2026-02-20T12:00:09.656 INFO:tasks.rados.rados.0.trial153.stdout:233: left oid 35 (ObjNum 140 snap 21 seq_num 140) 2026-02-20T12:00:09.684 INFO:tasks.rados.rados.0.trial153.stdout:233: done (5 left) 2026-02-20T12:00:09.684 INFO:tasks.rados.rados.0.trial153.stdout:235: done (4 left) 2026-02-20T12:00:09.684 INFO:tasks.rados.rados.0.trial153.stdout:236: done (3 left) 2026-02-20T12:00:09.685 INFO:tasks.rados.rados.0.trial153.stdout:239: done (2 left) 2026-02-20T12:00:09.685 INFO:tasks.rados.rados.0.trial153.stdout:240: write oid 37 current snap is 21 2026-02-20T12:00:09.685 INFO:tasks.rados.rados.0.trial153.stdout:240: seq_num 143 ranges {403401=454961,1584925=506039} 2026-02-20T12:00:09.688 INFO:tasks.rados.rados.0.trial153.stdout:240: writing trial15342161-37 from 403401 to 858362 tid 1 2026-02-20T12:00:09.694 INFO:tasks.rados.rados.0.trial153.stdout:240: writing trial15342161-37 from 1584925 to 2090964 tid 2 2026-02-20T12:00:09.694 INFO:tasks.rados.rados.0.trial153.stdout:241: snap_remove snap 19 2026-02-20T12:00:09.694 INFO:tasks.rados.rados.0.trial153.stdout:238: expect (ObjNum 123 snap 15 seq_num 123) 2026-02-20T12:00:09.694 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:09.714 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 646943 2026-02-20T12:00:09.726 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:09.963 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:10.040 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:10.034+0000 7f89e7c8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:10.043 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:10.038+0000 7f89e7c8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:10.068 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:10.084 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:10.650 INFO:tasks.rados.rados.0.trial153.stdout:238: done (3 left) 2026-02-20T12:00:10.650 INFO:tasks.rados.rados.0.trial153.stdout:241: done (2 left) 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:242: read oid 18 snap -1 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:242: expect (ObjNum 17 snap 0 seq_num 17) 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:243: snap_create 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:237: finishing write tid 1 to trial15342161-25 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:240: finishing write tid 1 to trial15342161-37 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:237: finishing write tid 2 to trial15342161-25 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:240: finishing write tid 2 to trial15342161-37 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:237: finishing write tid 3 to trial15342161-25 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:237: finishing write tid 4 to trial15342161-25 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:237: oid 25 updating version 0 to 29 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:237: oid 25 version 29 is already newer than 29 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:237: oid 25 version 29 is already newer than 29 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 25 v 29 (ObjNum 141 snap 21 seq_num 141) dirty exists 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:237: left oid 25 (ObjNum 141 snap 21 seq_num 141) 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:240: finishing write tid 3 to trial15342161-37 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:240: finishing write tid 4 to trial15342161-37 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:240: oid 37 updating version 0 to 29 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:240: oid 37 version 29 is already newer than 29 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:240: oid 37 version 29 is already newer than 29 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 29 (ObjNum 143 snap 21 seq_num 143) dirty exists 2026-02-20T12:00:10.651 INFO:tasks.rados.rados.0.trial153.stdout:240: left oid 37 (ObjNum 143 snap 21 seq_num 143) 2026-02-20T12:00:10.678 INFO:tasks.rados.rados.0.trial153.stdout:242: expect (ObjNum 17 snap 0 seq_num 17) 2026-02-20T12:00:10.704 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:11.492 INFO:tasks.rados.rados.0.trial153.stdout:237: done (3 left) 2026-02-20T12:00:11.492 INFO:tasks.rados.rados.0.trial153.stdout:240: done (2 left) 2026-02-20T12:00:11.492 INFO:tasks.rados.rados.0.trial153.stdout:242: done (1 left) 2026-02-20T12:00:11.492 INFO:tasks.rados.rados.0.trial153.stdout:243: done (0 left) 2026-02-20T12:00:11.492 INFO:tasks.rados.rados.0.trial153.stdout:244: snap_remove snap 21 2026-02-20T12:00:12.085 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:12.164 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:12.158+0000 7fb424d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:12.166 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:12.162+0000 7fb41f577640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:12.193 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:12.209 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:12.493 INFO:tasks.rados.rados.0.trial153.stdout:244: done (0 left) 2026-02-20T12:00:12.493 INFO:tasks.rados.rados.0.trial153.stdout:245: snap_create 2026-02-20T12:00:13.223 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:13.294 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:13.297 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:13.495 INFO:tasks.rados.rados.0.trial153.stdout:245: done (0 left) 2026-02-20T12:00:13.495 INFO:tasks.rados.rados.0.trial153.stdout:246: read oid 34 snap -1 2026-02-20T12:00:13.495 INFO:tasks.rados.rados.0.trial153.stdout:246: expect (ObjNum 103 snap 13 seq_num 103) 2026-02-20T12:00:13.496 INFO:tasks.rados.rados.0.trial153.stdout:247: snap_create 2026-02-20T12:00:13.539 INFO:tasks.rados.rados.0.trial153.stdout:246: expect (ObjNum 103 snap 13 seq_num 103) 2026-02-20T12:00:13.592 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:13.643 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:14.210 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:14.288 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:14.281+0000 7f575930b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:14.290 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:14.285+0000 7f575930b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:14.316 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:14.332 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:14.332 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:14.348 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~61s 2026-02-20T12:00:14.496 INFO:tasks.rados.rados.0.trial153.stdout:246: done (1 left) 2026-02-20T12:00:14.496 INFO:tasks.rados.rados.0.trial153.stdout:247: done (0 left) 2026-02-20T12:00:14.497 INFO:tasks.rados.rados.0.trial153.stdout:248: read oid 2 snap -1 2026-02-20T12:00:14.497 INFO:tasks.rados.rados.0.trial153.stdout:248: expect deleted 2026-02-20T12:00:14.497 INFO:tasks.rados.rados.0.trial153.stdout:249: delete oid 41 current snap is 24 2026-02-20T12:00:14.536 INFO:tasks.rados.rados.0.trial153.stdout:248: done (1 left) 2026-02-20T12:00:14.537 INFO:tasks.rados.rados.0.trial153.stdout:249: done (0 left) 2026-02-20T12:00:14.537 INFO:tasks.rados.rados.0.trial153.stdout:250: snap_create 2026-02-20T12:00:15.502 INFO:tasks.rados.rados.0.trial153.stdout:250: done (0 left) 2026-02-20T12:00:15.502 INFO:tasks.rados.rados.0.trial153.stdout:251: delete oid 35 current snap is 25 2026-02-20T12:00:15.539 INFO:tasks.rados.rados.0.trial153.stdout:251: done (0 left) 2026-02-20T12:00:15.539 INFO:tasks.rados.rados.0.trial153.stdout:252: delete oid 46 current snap is 25 2026-02-20T12:00:15.540 INFO:tasks.rados.rados.0.trial153.stdout:252: done (0 left) 2026-02-20T12:00:15.540 INFO:tasks.rados.rados.0.trial153.stdout:253: write oid 26 current snap is 25 2026-02-20T12:00:15.540 INFO:tasks.rados.rados.0.trial153.stdout:253: seq_num 147 ranges {470629=472310,1418853=513377} 2026-02-20T12:00:15.544 INFO:tasks.rados.rados.0.trial153.stdout:253: writing trial15342161-26 from 470629 to 942939 tid 1 2026-02-20T12:00:15.548 INFO:tasks.rados.rados.0.trial153.stdout:253: writing trial15342161-26 from 1418853 to 1932230 tid 2 2026-02-20T12:00:15.548 INFO:tasks.rados.rados.0.trial153.stdout:254: snap_remove snap 24 2026-02-20T12:00:16.333 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:16.411 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:16.405+0000 7f9806fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:16.414 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:16.409+0000 7f9806fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:16.440 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:16.455 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:16.493 INFO:tasks.rados.rados.0.trial153.stdout:254: done (1 left) 2026-02-20T12:00:16.493 INFO:tasks.rados.rados.0.trial153.stdout:255: snap_create 2026-02-20T12:00:16.493 INFO:tasks.rados.rados.0.trial153.stdout:253: finishing write tid 1 to trial15342161-26 2026-02-20T12:00:16.493 INFO:tasks.rados.rados.0.trial153.stdout:253: finishing write tid 2 to trial15342161-26 2026-02-20T12:00:16.493 INFO:tasks.rados.rados.0.trial153.stdout:253: finishing write tid 3 to trial15342161-26 2026-02-20T12:00:16.493 INFO:tasks.rados.rados.0.trial153.stdout:253: finishing write tid 4 to trial15342161-26 2026-02-20T12:00:16.493 INFO:tasks.rados.rados.0.trial153.stdout:253: oid 26 updating version 0 to 44 2026-02-20T12:00:16.493 INFO:tasks.rados.rados.0.trial153.stdout:253: oid 26 version 44 is already newer than 44 2026-02-20T12:00:16.494 INFO:tasks.rados.rados.0.trial153.stdout:253: oid 26 version 44 is already newer than 44 2026-02-20T12:00:16.494 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 26 v 44 (ObjNum 147 snap 25 seq_num 147) dirty exists 2026-02-20T12:00:16.494 INFO:tasks.rados.rados.0.trial153.stdout:253: left oid 26 (ObjNum 147 snap 25 seq_num 147) 2026-02-20T12:00:17.495 INFO:tasks.rados.rados.0.trial153.stdout:253: done (1 left) 2026-02-20T12:00:17.495 INFO:tasks.rados.rados.0.trial153.stdout:255: done (0 left) 2026-02-20T12:00:17.495 INFO:tasks.rados.rados.0.trial153.stdout:256: write (excl) oid 29 current snap is 26 2026-02-20T12:00:17.496 INFO:tasks.rados.rados.0.trial153.stdout:256: seq_num 148 ranges {487436=547559,1760982=657035} 2026-02-20T12:00:17.499 INFO:tasks.rados.rados.0.trial153.stdout:256: writing trial15342161-29 from 487436 to 1034995 tid 1 2026-02-20T12:00:17.505 INFO:tasks.rados.rados.0.trial153.stdout:256: writing trial15342161-29 from 1760982 to 2418017 tid 2 2026-02-20T12:00:17.505 INFO:tasks.rados.rados.0.trial153.stdout:257: snap_remove snap 23 2026-02-20T12:00:18.298 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:18.371 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:18.374 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:18.374 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:18.456 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:18.503 INFO:tasks.rados.rados.0.trial153.stdout:257: done (1 left) 2026-02-20T12:00:18.503 INFO:tasks.rados.rados.0.trial153.stdout:258: write (excl) oid 28 current snap is 26 2026-02-20T12:00:18.503 INFO:tasks.rados.rados.0.trial153.stdout:256: finishing write tid 1 to trial15342161-29 2026-02-20T12:00:18.503 INFO:tasks.rados.rados.0.trial153.stdout:258: seq_num 149 ranges {504243=622808,1819464=517046} 2026-02-20T12:00:18.507 INFO:tasks.rados.rados.0.trial153.stdout:258: writing trial15342161-28 from 504243 to 1127051 tid 1 2026-02-20T12:00:18.512 INFO:tasks.rados.rados.0.trial153.stdout:258: writing trial15342161-28 from 1819464 to 2336510 tid 2 2026-02-20T12:00:18.512 INFO:tasks.rados.rados.0.trial153.stdout:259: write oid 19 current snap is 26 2026-02-20T12:00:18.513 INFO:tasks.rados.rados.0.trial153.stdout:256: finishing write tid 2 to trial15342161-29 2026-02-20T12:00:18.513 INFO:tasks.rados.rados.0.trial153.stdout:259: seq_num 150 ranges {521050=698057,1877946=643104} 2026-02-20T12:00:18.517 INFO:tasks.rados.rados.0.trial153.stdout:259: writing trial15342161-19 from 521050 to 1219107 tid 1 2026-02-20T12:00:18.522 INFO:tasks.rados.rados.0.trial153.stdout:259: writing trial15342161-19 from 1877946 to 2521050 tid 2 2026-02-20T12:00:18.522 INFO:tasks.rados.rados.0.trial153.stdout:260: read oid 41 snap 22 2026-02-20T12:00:18.522 INFO:tasks.rados.rados.0.trial153.stdout:260: expect (ObjNum 94 snap 11 seq_num 94) 2026-02-20T12:00:18.523 INFO:tasks.rados.rados.0.trial153.stdout:261: read oid 39 snap -1 2026-02-20T12:00:18.523 INFO:tasks.rados.rados.0.trial153.stdout:261: expect (ObjNum 115 snap 13 seq_num 115) 2026-02-20T12:00:18.523 INFO:tasks.rados.rados.0.trial153.stdout:262: read oid 31 snap -1 2026-02-20T12:00:18.523 INFO:tasks.rados.rados.0.trial153.stdout:262: expect (ObjNum 128 snap 19 seq_num 128) 2026-02-20T12:00:18.523 INFO:tasks.rados.rados.0.trial153.stdout:263: write (excl) oid 1 current snap is 26 2026-02-20T12:00:18.523 INFO:tasks.rados.rados.0.trial153.stdout:256: finishing write tid 3 to trial15342161-29 2026-02-20T12:00:18.523 INFO:tasks.rados.rados.0.trial153.stdout:263: seq_num 151 ranges {537857=773306,1936428=520715} 2026-02-20T12:00:18.527 INFO:tasks.rados.rados.0.trial153.stdout:263: writing trial15342161-1 from 537857 to 1311163 tid 1 2026-02-20T12:00:18.531 INFO:tasks.rados.rados.0.trial153.stdout:263: writing trial15342161-1 from 1936428 to 2457143 tid 2 2026-02-20T12:00:18.531 INFO:tasks.rados.rados.0.trial153.stdout:264: snap_create 2026-02-20T12:00:18.531 INFO:tasks.rados.rados.0.trial153.stdout:256: finishing write tid 4 to trial15342161-29 2026-02-20T12:00:18.532 INFO:tasks.rados.rados.0.trial153.stdout:256: oid 29 updating version 0 to 76 2026-02-20T12:00:18.532 INFO:tasks.rados.rados.0.trial153.stdout:256: oid 29 updating version 76 to 78 2026-02-20T12:00:18.532 INFO:tasks.rados.rados.0.trial153.stdout:256: oid 29 version 78 is already newer than 78 2026-02-20T12:00:18.532 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 29 v 78 (ObjNum 148 snap 26 seq_num 148) dirty exists 2026-02-20T12:00:18.532 INFO:tasks.rados.rados.0.trial153.stdout:256: left oid 29 (ObjNum 148 snap 26 seq_num 148) 2026-02-20T12:00:18.543 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:18.537+0000 7f58303dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:18.545 INFO:tasks.rados.rados.0.trial153.stdout:262: expect (ObjNum 128 snap 19 seq_num 128) 2026-02-20T12:00:18.545 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:18.546 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:18.542+0000 7f58303dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:18.571 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:18.585 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:18.595 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:18.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:18.621 INFO:tasks.rados.rados.0.trial153.stdout:258: finishing write tid 1 to trial15342161-28 2026-02-20T12:00:18.621 INFO:tasks.rados.rados.0.trial153.stdout:261: expect (ObjNum 115 snap 13 seq_num 115) 2026-02-20T12:00:18.633 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 3931353 2026-02-20T12:00:18.680 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2676398 2026-02-20T12:00:18.690 INFO:tasks.rados.rados.0.trial153.stdout:260: expect (ObjNum 94 snap 11 seq_num 94) 2026-02-20T12:00:18.720 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:18.751 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 120366 2026-02-20T12:00:18.781 INFO:tasks.rados.rados.0.trial153.stdout:258: finishing write tid 2 to trial15342161-28 2026-02-20T12:00:18.781 INFO:tasks.rados.rados.0.trial153.stdout:263: finishing write tid 1 to trial15342161-1 2026-02-20T12:00:18.781 INFO:tasks.rados.rados.0.trial153.stdout:258: finishing write tid 3 to trial15342161-28 2026-02-20T12:00:18.781 INFO:tasks.rados.rados.0.trial153.stdout:258: finishing write tid 4 to trial15342161-28 2026-02-20T12:00:18.781 INFO:tasks.rados.rados.0.trial153.stdout:258: oid 28 updating version 0 to 63 2026-02-20T12:00:18.781 INFO:tasks.rados.rados.0.trial153.stdout:258: oid 28 version 63 is already newer than 63 2026-02-20T12:00:18.781 INFO:tasks.rados.rados.0.trial153.stdout:258: oid 28 version 63 is already newer than 61 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 28 v 63 (ObjNum 149 snap 26 seq_num 149) dirty exists 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:258: left oid 28 (ObjNum 149 snap 26 seq_num 149) 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:263: finishing write tid 2 to trial15342161-1 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:263: finishing write tid 3 to trial15342161-1 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:263: finishing write tid 4 to trial15342161-1 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:263: oid 1 updating version 0 to 82 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:263: oid 1 updating version 82 to 84 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:263: oid 1 version 84 is already newer than 84 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 1 v 84 (ObjNum 151 snap 26 seq_num 151) dirty exists 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:263: left oid 1 (ObjNum 151 snap 26 seq_num 151) 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:259: finishing write tid 1 to trial15342161-19 2026-02-20T12:00:18.782 INFO:tasks.rados.rados.0.trial153.stdout:259: finishing write tid 2 to trial15342161-19 2026-02-20T12:00:18.783 INFO:tasks.rados.rados.0.trial153.stdout:259: finishing write tid 3 to trial15342161-19 2026-02-20T12:00:18.783 INFO:tasks.rados.rados.0.trial153.stdout:259: finishing write tid 4 to trial15342161-19 2026-02-20T12:00:18.783 INFO:tasks.rados.rados.0.trial153.stdout:259: oid 19 updating version 0 to 40 2026-02-20T12:00:18.783 INFO:tasks.rados.rados.0.trial153.stdout:259: oid 19 version 40 is already newer than 40 2026-02-20T12:00:18.783 INFO:tasks.rados.rados.0.trial153.stdout:259: oid 19 version 40 is already newer than 40 2026-02-20T12:00:18.783 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 40 (ObjNum 150 snap 26 seq_num 150) dirty exists 2026-02-20T12:00:18.783 INFO:tasks.rados.rados.0.trial153.stdout:259: left oid 19 (ObjNum 150 snap 26 seq_num 150) 2026-02-20T12:00:19.507 INFO:tasks.rados.rados.0.trial153.stdout:256: done (7 left) 2026-02-20T12:00:19.507 INFO:tasks.rados.rados.0.trial153.stdout:258: done (6 left) 2026-02-20T12:00:19.508 INFO:tasks.rados.rados.0.trial153.stdout:259: done (5 left) 2026-02-20T12:00:19.508 INFO:tasks.rados.rados.0.trial153.stdout:260: done (4 left) 2026-02-20T12:00:19.508 INFO:tasks.rados.rados.0.trial153.stdout:261: done (3 left) 2026-02-20T12:00:19.508 INFO:tasks.rados.rados.0.trial153.stdout:262: done (2 left) 2026-02-20T12:00:19.508 INFO:tasks.rados.rados.0.trial153.stdout:263: done (1 left) 2026-02-20T12:00:19.508 INFO:tasks.rados.rados.0.trial153.stdout:264: done (0 left) 2026-02-20T12:00:19.508 INFO:tasks.rados.rados.0.trial153.stdout:265: write oid 41 current snap is 27 2026-02-20T12:00:19.508 INFO:tasks.rados.rados.0.trial153.stdout:265: seq_num 152 ranges {554664=448555,1478557=664373} 2026-02-20T12:00:19.511 INFO:tasks.rados.rados.0.trial153.stdout:265: writing trial15342161-41 from 554664 to 1003219 tid 1 2026-02-20T12:00:19.515 INFO:tasks.rados.rados.0.trial153.stdout:265: writing trial15342161-41 from 1478557 to 2142930 tid 2 2026-02-20T12:00:19.516 INFO:tasks.rados.rados.0.trial153.stdout:266: snap_remove snap 26 2026-02-20T12:00:19.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~66s 2026-02-20T12:00:20.507 INFO:tasks.rados.rados.0.trial153.stdout:266: done (1 left) 2026-02-20T12:00:20.508 INFO:tasks.rados.rados.0.trial153.stdout:267: snap_create 2026-02-20T12:00:20.508 INFO:tasks.rados.rados.0.trial153.stdout:265: finishing write tid 1 to trial15342161-41 2026-02-20T12:00:20.508 INFO:tasks.rados.rados.0.trial153.stdout:265: finishing write tid 2 to trial15342161-41 2026-02-20T12:00:20.508 INFO:tasks.rados.rados.0.trial153.stdout:265: finishing write tid 3 to trial15342161-41 2026-02-20T12:00:20.508 INFO:tasks.rados.rados.0.trial153.stdout:265: finishing write tid 4 to trial15342161-41 2026-02-20T12:00:20.508 INFO:tasks.rados.rados.0.trial153.stdout:265: oid 41 updating version 0 to 40 2026-02-20T12:00:20.508 INFO:tasks.rados.rados.0.trial153.stdout:265: oid 41 version 40 is already newer than 40 2026-02-20T12:00:20.508 INFO:tasks.rados.rados.0.trial153.stdout:265: oid 41 version 40 is already newer than 40 2026-02-20T12:00:20.508 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 41 v 40 (ObjNum 152 snap 27 seq_num 152) dirty exists 2026-02-20T12:00:20.508 INFO:tasks.rados.rados.0.trial153.stdout:265: left oid 41 (ObjNum 152 snap 27 seq_num 152) 2026-02-20T12:00:20.605 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:20.681 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:20.675+0000 7f489921a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:20.683 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:20.679+0000 7f489921a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:20.710 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:20.725 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:20.726 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:21.510 INFO:tasks.rados.rados.0.trial153.stdout:265: done (1 left) 2026-02-20T12:00:21.511 INFO:tasks.rados.rados.0.trial153.stdout:267: done (0 left) 2026-02-20T12:00:21.511 INFO:tasks.rados.rados.0.trial153.stdout:268: delete oid 33 current snap is 28 2026-02-20T12:00:21.545 INFO:tasks.rados.rados.0.trial153.stdout:268: done (0 left) 2026-02-20T12:00:21.545 INFO:tasks.rados.rados.0.trial153.stdout:269: snap_create 2026-02-20T12:00:22.519 INFO:tasks.rados.rados.0.trial153.stdout:269: done (0 left) 2026-02-20T12:00:22.519 INFO:tasks.rados.rados.0.trial153.stdout:270: write (excl) oid 13 current snap is 29 2026-02-20T12:00:22.519 INFO:tasks.rados.rados.0.trial153.stdout:270: seq_num 154 ranges {588278=715406,1711874=668042} 2026-02-20T12:00:22.524 INFO:tasks.rados.rados.0.trial153.stdout:270: writing trial15342161-13 from 588278 to 1303684 tid 1 2026-02-20T12:00:22.528 INFO:tasks.rados.rados.0.trial153.stdout:270: writing trial15342161-13 from 1711874 to 2379916 tid 2 2026-02-20T12:00:22.528 INFO:tasks.rados.rados.0.trial153.stdout:271: snap_create 2026-02-20T12:00:22.571 INFO:tasks.rados.rados.0.trial153.stdout:270: finishing write tid 1 to trial15342161-13 2026-02-20T12:00:22.585 INFO:tasks.rados.rados.0.trial153.stdout:270: finishing write tid 2 to trial15342161-13 2026-02-20T12:00:22.590 INFO:tasks.rados.rados.0.trial153.stdout:270: finishing write tid 3 to trial15342161-13 2026-02-20T12:00:22.592 INFO:tasks.rados.rados.0.trial153.stdout:270: finishing write tid 4 to trial15342161-13 2026-02-20T12:00:22.592 INFO:tasks.rados.rados.0.trial153.stdout:270: oid 13 updating version 0 to 88 2026-02-20T12:00:22.592 INFO:tasks.rados.rados.0.trial153.stdout:270: oid 13 version 88 is already newer than 86 2026-02-20T12:00:22.592 INFO:tasks.rados.rados.0.trial153.stdout:270: oid 13 version 88 is already newer than 88 2026-02-20T12:00:22.592 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 13 v 88 (ObjNum 154 snap 29 seq_num 154) dirty exists 2026-02-20T12:00:22.592 INFO:tasks.rados.rados.0.trial153.stdout:270: left oid 13 (ObjNum 154 snap 29 seq_num 154) 2026-02-20T12:00:22.726 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:22.803 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:22.798+0000 7fd3a70c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:22.806 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:22.801+0000 7fd3a70c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:22.833 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:22.849 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:23.375 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:23.448 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:23.452 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:23.523 INFO:tasks.rados.rados.0.trial153.stdout:270: done (1 left) 2026-02-20T12:00:23.523 INFO:tasks.rados.rados.0.trial153.stdout:271: done (0 left) 2026-02-20T12:00:23.523 INFO:tasks.rados.rados.0.trial153.stdout:272: read oid 49 snap -1 2026-02-20T12:00:23.523 INFO:tasks.rados.rados.0.trial153.stdout:272: expect (ObjNum 108 snap 13 seq_num 108) 2026-02-20T12:00:23.523 INFO:tasks.rados.rados.0.trial153.stdout:273: snap_create 2026-02-20T12:00:23.560 INFO:tasks.rados.rados.0.trial153.stdout:272: expect (ObjNum 108 snap 13 seq_num 108) 2026-02-20T12:00:23.560 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:23.652 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 449432 2026-02-20T12:00:24.525 INFO:tasks.rados.rados.0.trial153.stdout:272: done (1 left) 2026-02-20T12:00:24.525 INFO:tasks.rados.rados.0.trial153.stdout:273: done (0 left) 2026-02-20T12:00:24.525 INFO:tasks.rados.rados.0.trial153.stdout:274: read oid 20 snap -1 2026-02-20T12:00:24.526 INFO:tasks.rados.rados.0.trial153.stdout:274: expect (ObjNum 126 snap 16 seq_num 126) 2026-02-20T12:00:24.526 INFO:tasks.rados.rados.0.trial153.stdout:275: snap_remove snap 30 2026-02-20T12:00:24.850 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:24.926 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:24.920+0000 7fb431774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:24.929 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:24.924+0000 7fb431774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:24.957 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:24.972 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:24.973 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:25.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~72s 2026-02-20T12:00:25.531 INFO:tasks.rados.rados.0.trial153.stdout:275: done (1 left) 2026-02-20T12:00:25.531 INFO:tasks.rados.rados.0.trial153.stdout:276: snap_remove snap 29 2026-02-20T12:00:26.496 INFO:tasks.rados.rados.0.trial153.stdout:276: done (1 left) 2026-02-20T12:00:26.496 INFO:tasks.rados.rados.0.trial153.stdout:277: delete oid 21 current snap is 31 2026-02-20T12:00:26.496 INFO:tasks.rados.rados.0.trial153.stdout:274: expect (ObjNum 126 snap 16 seq_num 126) 2026-02-20T12:00:26.496 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:26.547 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1147522 2026-02-20T12:00:26.558 INFO:tasks.rados.rados.0.trial153.stdout:274: done (1 left) 2026-02-20T12:00:26.558 INFO:tasks.rados.rados.0.trial153.stdout:277: done (0 left) 2026-02-20T12:00:26.558 INFO:tasks.rados.rados.0.trial153.stdout:278: write (excl) oid 24 current snap is 31 2026-02-20T12:00:26.559 INFO:tasks.rados.rados.0.trial153.stdout:278: seq_num 156 ranges {621892=465904,1712485=671711} 2026-02-20T12:00:26.562 INFO:tasks.rados.rados.0.trial153.stdout:278: writing trial15342161-24 from 621892 to 1087796 tid 1 2026-02-20T12:00:26.568 INFO:tasks.rados.rados.0.trial153.stdout:278: writing trial15342161-24 from 1712485 to 2384196 tid 2 2026-02-20T12:00:26.568 INFO:tasks.rados.rados.0.trial153.stdout:279: read oid 7 snap -1 2026-02-20T12:00:26.568 INFO:tasks.rados.rados.0.trial153.stdout:279: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T12:00:26.568 INFO:tasks.rados.rados.0.trial153.stdout:280: read oid 45 snap -1 2026-02-20T12:00:26.568 INFO:tasks.rados.rados.0.trial153.stdout:280: expect (ObjNum 136 snap 20 seq_num 136) 2026-02-20T12:00:26.568 INFO:tasks.rados.rados.0.trial153.stdout:281: snap_remove snap 22 2026-02-20T12:00:26.974 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:27.050 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:27.044+0000 7f9010fdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:27.053 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:27.048+0000 7f9010fdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:27.079 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:27.095 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:27.096 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:27.535 INFO:tasks.rados.rados.0.trial153.stdout:281: done (3 left) 2026-02-20T12:00:27.535 INFO:tasks.rados.rados.0.trial153.stdout:282: delete oid 36 current snap is 31 2026-02-20T12:00:27.535 INFO:tasks.rados.rados.0.trial153.stdout:279: expect (ObjNum 63 snap 3 seq_num 63) 2026-02-20T12:00:27.535 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:27.572 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 360257 2026-02-20T12:00:27.612 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1777147 2026-02-20T12:00:27.630 INFO:tasks.rados.rados.0.trial153.stdout:278: finishing write tid 1 to trial15342161-24 2026-02-20T12:00:27.630 INFO:tasks.rados.rados.0.trial153.stdout:280: expect (ObjNum 136 snap 20 seq_num 136) 2026-02-20T12:00:27.666 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1940025 2026-02-20T12:00:27.699 INFO:tasks.rados.rados.0.trial153.stdout:278: finishing write tid 2 to trial15342161-24 2026-02-20T12:00:27.699 INFO:tasks.rados.rados.0.trial153.stdout:278: finishing write tid 3 to trial15342161-24 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:278: finishing write tid 4 to trial15342161-24 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:278: oid 24 updating version 0 to 13 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:278: oid 24 version 13 is already newer than 11 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:278: oid 24 version 13 is already newer than 13 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 24 v 13 (ObjNum 156 snap 31 seq_num 156) dirty exists 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:278: left oid 24 (ObjNum 156 snap 31 seq_num 156) 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:278: done (3 left) 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:279: done (2 left) 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:280: done (1 left) 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:282: done (0 left) 2026-02-20T12:00:27.700 INFO:tasks.rados.rados.0.trial153.stdout:283: snap_create 2026-02-20T12:00:28.452 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:28.523 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:28.527 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:28.538 INFO:tasks.rados.rados.0.trial153.stdout:283: done (0 left) 2026-02-20T12:00:28.538 INFO:tasks.rados.rados.0.trial153.stdout:284: snap_create 2026-02-20T12:00:29.096 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:29.175 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:29.169+0000 7f0d616b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:29.177 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:29.173+0000 7f0d616b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:29.204 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:29.220 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:29.221 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:29.543 INFO:tasks.rados.rados.0.trial153.stdout:284: done (0 left) 2026-02-20T12:00:29.543 INFO:tasks.rados.rados.0.trial153.stdout:285: snap_remove snap 32 2026-02-20T12:00:30.545 INFO:tasks.rados.rados.0.trial153.stdout:285: done (0 left) 2026-02-20T12:00:30.546 INFO:tasks.rados.rados.0.trial153.stdout:286: snap_create 2026-02-20T12:00:30.861 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~77s 2026-02-20T12:00:31.222 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:31.300 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:31.294+0000 7feec67b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:31.303 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:31.298+0000 7feec67b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:31.329 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:31.346 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:31.551 INFO:tasks.rados.rados.0.trial153.stdout:286: done (0 left) 2026-02-20T12:00:31.552 INFO:tasks.rados.rados.0.trial153.stdout:287: write (excl) oid 4 current snap is 34 2026-02-20T12:00:31.552 INFO:tasks.rados.rados.0.trial153.stdout:287: seq_num 158 ranges {655506=616402,1829449=675380} 2026-02-20T12:00:31.556 INFO:tasks.rados.rados.0.trial153.stdout:287: writing trial15342161-4 from 655506 to 1271908 tid 1 2026-02-20T12:00:31.562 INFO:tasks.rados.rados.0.trial153.stdout:287: writing trial15342161-4 from 1829449 to 2504829 tid 2 2026-02-20T12:00:31.562 INFO:tasks.rados.rados.0.trial153.stdout:288: write (excl) oid 22 current snap is 34 2026-02-20T12:00:31.562 INFO:tasks.rados.rados.0.trial153.stdout:288: seq_num 159 ranges {672313=691651,1887931=419038} 2026-02-20T12:00:31.567 INFO:tasks.rados.rados.0.trial153.stdout:288: writing trial15342161-22 from 672313 to 1363964 tid 1 2026-02-20T12:00:31.570 INFO:tasks.rados.rados.0.trial153.stdout:288: writing trial15342161-22 from 1887931 to 2306969 tid 2 2026-02-20T12:00:31.571 INFO:tasks.rados.rados.0.trial153.stdout:289: write oid 34 current snap is 34 2026-02-20T12:00:31.571 INFO:tasks.rados.rados.0.trial153.stdout:289: seq_num 160 ranges {689120=483253,1946413=679049} 2026-02-20T12:00:31.575 INFO:tasks.rados.rados.0.trial153.stdout:289: writing trial15342161-34 from 689120 to 1172373 tid 1 2026-02-20T12:00:31.581 INFO:tasks.rados.rados.0.trial153.stdout:289: writing trial15342161-34 from 1946413 to 2625462 tid 2 2026-02-20T12:00:31.581 INFO:tasks.rados.rados.0.trial153.stdout:290: write oid 15 current snap is 34 2026-02-20T12:00:31.581 INFO:tasks.rados.rados.0.trial153.stdout:290: seq_num 161 ranges {705927=558502,2004895=422707} 2026-02-20T12:00:31.585 INFO:tasks.rados.rados.0.trial153.stdout:290: writing trial15342161-15 from 705927 to 1264429 tid 1 2026-02-20T12:00:31.590 INFO:tasks.rados.rados.0.trial153.stdout:290: writing trial15342161-15 from 2004895 to 2427602 tid 2 2026-02-20T12:00:31.590 INFO:tasks.rados.rados.0.trial153.stdout:291: read oid 10 snap 33 2026-02-20T12:00:31.590 INFO:tasks.rados.rados.0.trial153.stdout:291: expect deleted 2026-02-20T12:00:31.590 INFO:tasks.rados.rados.0.trial153.stdout:292: delete oid 11 current snap is 34 2026-02-20T12:00:31.594 INFO:tasks.rados.rados.0.trial153.stdout:287: finishing write tid 1 to trial15342161-4 2026-02-20T12:00:31.620 INFO:tasks.rados.rados.0.trial153.stdout:287: finishing write tid 2 to trial15342161-4 2026-02-20T12:00:31.631 INFO:tasks.rados.rados.0.trial153.stdout:288: finishing write tid 1 to trial15342161-22 2026-02-20T12:00:31.640 INFO:tasks.rados.rados.0.trial153.stdout:287: finishing write tid 3 to trial15342161-4 2026-02-20T12:00:31.643 INFO:tasks.rados.rados.0.trial153.stdout:287: finishing write tid 4 to trial15342161-4 2026-02-20T12:00:31.643 INFO:tasks.rados.rados.0.trial153.stdout:287: oid 4 updating version 0 to 56 2026-02-20T12:00:31.643 INFO:tasks.rados.rados.0.trial153.stdout:287: oid 4 version 56 is already newer than 54 2026-02-20T12:00:31.643 INFO:tasks.rados.rados.0.trial153.stdout:287: oid 4 version 56 is already newer than 56 2026-02-20T12:00:31.643 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 4 v 56 (ObjNum 158 snap 34 seq_num 158) dirty exists 2026-02-20T12:00:31.643 INFO:tasks.rados.rados.0.trial153.stdout:287: left oid 4 (ObjNum 158 snap 34 seq_num 158) 2026-02-20T12:00:31.652 INFO:tasks.rados.rados.0.trial153.stdout:290: finishing write tid 1 to trial15342161-15 2026-02-20T12:00:31.655 INFO:tasks.rados.rados.0.trial153.stdout:287: done (5 left) 2026-02-20T12:00:31.655 INFO:tasks.rados.rados.0.trial153.stdout:291: done (4 left) 2026-02-20T12:00:31.655 INFO:tasks.rados.rados.0.trial153.stdout:292: done (3 left) 2026-02-20T12:00:31.656 INFO:tasks.rados.rados.0.trial153.stdout:293: delete oid 6 current snap is 34 2026-02-20T12:00:31.668 INFO:tasks.rados.rados.0.trial153.stdout:290: finishing write tid 2 to trial15342161-15 2026-02-20T12:00:31.679 INFO:tasks.rados.rados.0.trial153.stdout:290: finishing write tid 3 to trial15342161-15 2026-02-20T12:00:31.685 INFO:tasks.rados.rados.0.trial153.stdout:290: finishing write tid 4 to trial15342161-15 2026-02-20T12:00:31.685 INFO:tasks.rados.rados.0.trial153.stdout:290: oid 15 updating version 0 to 46 2026-02-20T12:00:31.685 INFO:tasks.rados.rados.0.trial153.stdout:290: oid 15 version 46 is already newer than 46 2026-02-20T12:00:31.685 INFO:tasks.rados.rados.0.trial153.stdout:290: oid 15 version 46 is already newer than 46 2026-02-20T12:00:31.685 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 15 v 46 (ObjNum 161 snap 34 seq_num 161) dirty exists 2026-02-20T12:00:31.686 INFO:tasks.rados.rados.0.trial153.stdout:290: left oid 15 (ObjNum 161 snap 34 seq_num 161) 2026-02-20T12:00:31.686 INFO:tasks.rados.rados.0.trial153.stdout:290: done (3 left) 2026-02-20T12:00:31.686 INFO:tasks.rados.rados.0.trial153.stdout:293: done (2 left) 2026-02-20T12:00:31.686 INFO:tasks.rados.rados.0.trial153.stdout:294: write (excl) oid 8 current snap is 34 2026-02-20T12:00:31.686 INFO:tasks.rados.rados.0.trial153.stdout:294: seq_num 164 ranges {756348=784249,2063988=686387} 2026-02-20T12:00:31.691 INFO:tasks.rados.rados.0.trial153.stdout:294: writing trial15342161-8 from 756348 to 1540597 tid 1 2026-02-20T12:00:31.696 INFO:tasks.rados.rados.0.trial153.stdout:294: writing trial15342161-8 from 2063988 to 2750375 tid 2 2026-02-20T12:00:31.696 INFO:tasks.rados.rados.0.trial153.stdout:295: snap_create 2026-02-20T12:00:31.696 INFO:tasks.rados.rados.0.trial153.stdout:289: finishing write tid 1 to trial15342161-34 2026-02-20T12:00:31.696 INFO:tasks.rados.rados.0.trial153.stdout:288: finishing write tid 2 to trial15342161-22 2026-02-20T12:00:31.697 INFO:tasks.rados.rados.0.trial153.stdout:289: finishing write tid 2 to trial15342161-34 2026-02-20T12:00:31.700 INFO:tasks.rados.rados.0.trial153.stdout:288: finishing write tid 3 to trial15342161-22 2026-02-20T12:00:31.702 INFO:tasks.rados.rados.0.trial153.stdout:288: finishing write tid 4 to trial15342161-22 2026-02-20T12:00:31.702 INFO:tasks.rados.rados.0.trial153.stdout:288: oid 22 updating version 0 to 65 2026-02-20T12:00:31.702 INFO:tasks.rados.rados.0.trial153.stdout:288: oid 22 updating version 65 to 67 2026-02-20T12:00:31.702 INFO:tasks.rados.rados.0.trial153.stdout:288: oid 22 version 67 is already newer than 67 2026-02-20T12:00:31.702 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 22 v 67 (ObjNum 159 snap 34 seq_num 159) dirty exists 2026-02-20T12:00:31.702 INFO:tasks.rados.rados.0.trial153.stdout:288: left oid 22 (ObjNum 159 snap 34 seq_num 159) 2026-02-20T12:00:31.704 INFO:tasks.rados.rados.0.trial153.stdout:289: finishing write tid 3 to trial15342161-34 2026-02-20T12:00:31.706 INFO:tasks.rados.rados.0.trial153.stdout:289: finishing write tid 4 to trial15342161-34 2026-02-20T12:00:31.706 INFO:tasks.rados.rados.0.trial153.stdout:289: oid 34 updating version 0 to 53 2026-02-20T12:00:31.706 INFO:tasks.rados.rados.0.trial153.stdout:289: oid 34 version 53 is already newer than 53 2026-02-20T12:00:31.706 INFO:tasks.rados.rados.0.trial153.stdout:289: oid 34 version 53 is already newer than 53 2026-02-20T12:00:31.706 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 53 (ObjNum 160 snap 34 seq_num 160) dirty exists 2026-02-20T12:00:31.706 INFO:tasks.rados.rados.0.trial153.stdout:289: left oid 34 (ObjNum 160 snap 34 seq_num 160) 2026-02-20T12:00:31.707 INFO:tasks.rados.rados.0.trial153.stdout:294: finishing write tid 1 to trial15342161-8 2026-02-20T12:00:31.719 INFO:tasks.rados.rados.0.trial153.stdout:294: finishing write tid 2 to trial15342161-8 2026-02-20T12:00:31.723 INFO:tasks.rados.rados.0.trial153.stdout:294: finishing write tid 3 to trial15342161-8 2026-02-20T12:00:31.724 INFO:tasks.rados.rados.0.trial153.stdout:294: finishing write tid 4 to trial15342161-8 2026-02-20T12:00:31.725 INFO:tasks.rados.rados.0.trial153.stdout:294: oid 8 updating version 0 to 32 2026-02-20T12:00:31.725 INFO:tasks.rados.rados.0.trial153.stdout:294: oid 8 version 32 is already newer than 30 2026-02-20T12:00:31.725 INFO:tasks.rados.rados.0.trial153.stdout:294: oid 8 version 32 is already newer than 32 2026-02-20T12:00:31.725 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 8 v 32 (ObjNum 164 snap 34 seq_num 164) dirty exists 2026-02-20T12:00:31.725 INFO:tasks.rados.rados.0.trial153.stdout:294: left oid 8 (ObjNum 164 snap 34 seq_num 164) 2026-02-20T12:00:32.554 INFO:tasks.rados.rados.0.trial153.stdout:288: done (3 left) 2026-02-20T12:00:32.554 INFO:tasks.rados.rados.0.trial153.stdout:289: done (2 left) 2026-02-20T12:00:32.554 INFO:tasks.rados.rados.0.trial153.stdout:294: done (1 left) 2026-02-20T12:00:32.554 INFO:tasks.rados.rados.0.trial153.stdout:295: done (0 left) 2026-02-20T12:00:32.555 INFO:tasks.rados.rados.0.trial153.stdout:296: write oid 33 current snap is 35 2026-02-20T12:00:32.555 INFO:tasks.rados.rados.0.trial153.stdout:296: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2026-02-20T12:00:32.558 INFO:tasks.rados.rados.0.trial153.stdout:296: writing trial15342161-33 from 773155 to 1232653 tid 1 2026-02-20T12:00:32.562 INFO:tasks.rados.rados.0.trial153.stdout:296: writing trial15342161-33 from 1722470 to 2152515 tid 2 2026-02-20T12:00:32.564 INFO:tasks.rados.rados.0.trial153.stdout:296: writing trial15342161-33 from 2648676 to 2773155 tid 3 2026-02-20T12:00:32.564 INFO:tasks.rados.rados.0.trial153.stdout:297: read oid 5 snap -1 2026-02-20T12:00:32.564 INFO:tasks.rados.rados.0.trial153.stdout:297: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-20T12:00:32.565 INFO:tasks.rados.rados.0.trial153.stdout:298: snap_create 2026-02-20T12:00:32.588 INFO:tasks.rados.rados.0.trial153.stdout:296: finishing write tid 1 to trial15342161-33 2026-02-20T12:00:32.593 INFO:tasks.rados.rados.0.trial153.stdout:297: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-20T12:00:32.593 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:32.611 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:32.646 INFO:tasks.rados.rados.0.trial153.stdout:296: finishing write tid 2 to trial15342161-33 2026-02-20T12:00:32.646 INFO:tasks.rados.rados.0.trial153.stdout:296: finishing write tid 3 to trial15342161-33 2026-02-20T12:00:32.646 INFO:tasks.rados.rados.0.trial153.stdout:296: finishing write tid 4 to trial15342161-33 2026-02-20T12:00:32.646 INFO:tasks.rados.rados.0.trial153.stdout:296: finishing write tid 5 to trial15342161-33 2026-02-20T12:00:32.646 INFO:tasks.rados.rados.0.trial153.stdout:296: oid 33 updating version 0 to 27 2026-02-20T12:00:32.646 INFO:tasks.rados.rados.0.trial153.stdout:296: oid 33 version 27 is already newer than 27 2026-02-20T12:00:32.646 INFO:tasks.rados.rados.0.trial153.stdout:296: oid 33 version 27 is already newer than 27 2026-02-20T12:00:32.646 INFO:tasks.rados.rados.0.trial153.stdout:296: oid 33 version 27 is already newer than 27 2026-02-20T12:00:32.646 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 33 v 27 (ObjNum 165 snap 35 seq_num 165) dirty exists 2026-02-20T12:00:32.646 INFO:tasks.rados.rados.0.trial153.stdout:296: left oid 33 (ObjNum 165 snap 35 seq_num 165) 2026-02-20T12:00:33.346 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:33.423 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:33.417+0000 7f296bded640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:33.425 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:33.421+0000 7f296bded640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:33.451 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:33.467 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:33.528 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:33.556 INFO:tasks.rados.rados.0.trial153.stdout:296: done (2 left) 2026-02-20T12:00:33.556 INFO:tasks.rados.rados.0.trial153.stdout:297: done (1 left) 2026-02-20T12:00:33.556 INFO:tasks.rados.rados.0.trial153.stdout:298: done (0 left) 2026-02-20T12:00:33.556 INFO:tasks.rados.rados.0.trial153.stdout:299: write oid 38 current snap is 36 2026-02-20T12:00:33.557 INFO:tasks.rados.rados.0.trial153.stdout:299: seq_num 166 ranges {789962=534747,1780952=690056} 2026-02-20T12:00:33.560 INFO:tasks.rados.rados.0.trial153.stdout:299: writing trial15342161-38 from 789962 to 1324709 tid 1 2026-02-20T12:00:33.566 INFO:tasks.rados.rados.0.trial153.stdout:299: writing trial15342161-38 from 1780952 to 2471008 tid 2 2026-02-20T12:00:33.566 INFO:tasks.rados.rados.0.trial153.stdout:300: read oid 40 snap -1 2026-02-20T12:00:33.566 INFO:tasks.rados.rados.0.trial153.stdout:300: expect (ObjNum 102 snap 13 seq_num 102) 2026-02-20T12:00:33.566 INFO:tasks.rados.rados.0.trial153.stdout:301: read oid 13 snap 33 2026-02-20T12:00:33.566 INFO:tasks.rados.rados.0.trial153.stdout:301: expect (ObjNum 154 snap 29 seq_num 154) 2026-02-20T12:00:33.566 INFO:tasks.rados.rados.0.trial153.stdout:302: read oid 30 snap -1 2026-02-20T12:00:33.566 INFO:tasks.rados.rados.0.trial153.stdout:302: expect deleted 2026-02-20T12:00:33.566 INFO:tasks.rados.rados.0.trial153.stdout:303: delete oid 27 current snap is 36 2026-02-20T12:00:33.591 INFO:tasks.rados.rados.0.trial153.stdout:300: expect (ObjNum 102 snap 13 seq_num 102) 2026-02-20T12:00:33.591 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 373701 2026-02-20T12:00:33.602 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:33.605 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:33.703 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1516895 2026-02-20T12:00:33.736 INFO:tasks.rados.rados.0.trial153.stdout:300: done (4 left) 2026-02-20T12:00:33.737 INFO:tasks.rados.rados.0.trial153.stdout:302: done (3 left) 2026-02-20T12:00:33.737 INFO:tasks.rados.rados.0.trial153.stdout:303: done (2 left) 2026-02-20T12:00:33.737 INFO:tasks.rados.rados.0.trial153.stdout:304: read oid 24 snap -1 2026-02-20T12:00:33.737 INFO:tasks.rados.rados.0.trial153.stdout:304: expect (ObjNum 156 snap 31 seq_num 156) 2026-02-20T12:00:33.737 INFO:tasks.rados.rados.0.trial153.stdout:305: snap_create 2026-02-20T12:00:33.737 INFO:tasks.rados.rados.0.trial153.stdout:299: finishing write tid 1 to trial15342161-38 2026-02-20T12:00:33.737 INFO:tasks.rados.rados.0.trial153.stdout:299: finishing write tid 2 to trial15342161-38 2026-02-20T12:00:33.737 INFO:tasks.rados.rados.0.trial153.stdout:299: finishing write tid 3 to trial15342161-38 2026-02-20T12:00:33.737 INFO:tasks.rados.rados.0.trial153.stdout:301: expect (ObjNum 154 snap 29 seq_num 154) 2026-02-20T12:00:33.831 INFO:tasks.rados.rados.0.trial153.stdout:299: finishing write tid 4 to trial15342161-38 2026-02-20T12:00:33.831 INFO:tasks.rados.rados.0.trial153.stdout:299: oid 38 updating version 0 to 60 2026-02-20T12:00:33.831 INFO:tasks.rados.rados.0.trial153.stdout:299: oid 38 version 60 is already newer than 60 2026-02-20T12:00:33.831 INFO:tasks.rados.rados.0.trial153.stdout:299: oid 38 version 60 is already newer than 60 2026-02-20T12:00:33.831 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 38 v 60 (ObjNum 166 snap 36 seq_num 166) dirty exists 2026-02-20T12:00:33.831 INFO:tasks.rados.rados.0.trial153.stdout:299: left oid 38 (ObjNum 166 snap 36 seq_num 166) 2026-02-20T12:00:33.831 INFO:tasks.rados.rados.0.trial153.stdout:304: expect (ObjNum 156 snap 31 seq_num 156) 2026-02-20T12:00:33.885 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:34.559 INFO:tasks.rados.rados.0.trial153.stdout:299: done (3 left) 2026-02-20T12:00:34.559 INFO:tasks.rados.rados.0.trial153.stdout:301: done (2 left) 2026-02-20T12:00:34.559 INFO:tasks.rados.rados.0.trial153.stdout:304: done (1 left) 2026-02-20T12:00:34.559 INFO:tasks.rados.rados.0.trial153.stdout:305: done (0 left) 2026-02-20T12:00:34.559 INFO:tasks.rados.rados.0.trial153.stdout:306: read oid 13 snap 34 2026-02-20T12:00:34.559 INFO:tasks.rados.rados.0.trial153.stdout:306: expect (ObjNum 154 snap 29 seq_num 154) 2026-02-20T12:00:34.559 INFO:tasks.rados.rados.0.trial153.stdout:307: snap_remove snap 33 2026-02-20T12:00:35.468 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:35.544 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:35.538+0000 7f654c05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:35.546 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:35.542+0000 7f654c05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:35.561 INFO:tasks.rados.rados.0.trial153.stdout:307: done (1 left) 2026-02-20T12:00:35.562 INFO:tasks.rados.rados.0.trial153.stdout:308: write (excl) oid 36 current snap is 37 2026-02-20T12:00:35.562 INFO:tasks.rados.rados.0.trial153.stdout:308: seq_num 168 ranges {423576=401598,1497916=693725} 2026-02-20T12:00:35.565 INFO:tasks.rados.rados.0.trial153.stdout:308: writing trial15342161-36 from 423576 to 825174 tid 1 2026-02-20T12:00:35.570 INFO:tasks.rados.rados.0.trial153.stdout:308: writing trial15342161-36 from 1497916 to 2191641 tid 2 2026-02-20T12:00:35.570 INFO:tasks.rados.rados.0.trial153.stdout:309: snap_create 2026-02-20T12:00:35.570 INFO:tasks.rados.rados.0.trial153.stdout:306: expect (ObjNum 154 snap 29 seq_num 154) 2026-02-20T12:00:35.587 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:35.603 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:35.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:35.640 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:35.675 INFO:tasks.rados.rados.0.trial153.stdout:308: finishing write tid 1 to trial15342161-36 2026-02-20T12:00:35.675 INFO:tasks.rados.rados.0.trial153.stdout:308: finishing write tid 2 to trial15342161-36 2026-02-20T12:00:35.675 INFO:tasks.rados.rados.0.trial153.stdout:308: finishing write tid 3 to trial15342161-36 2026-02-20T12:00:35.675 INFO:tasks.rados.rados.0.trial153.stdout:308: finishing write tid 4 to trial15342161-36 2026-02-20T12:00:35.675 INFO:tasks.rados.rados.0.trial153.stdout:308: oid 36 updating version 0 to 28 2026-02-20T12:00:35.675 INFO:tasks.rados.rados.0.trial153.stdout:308: oid 36 updating version 28 to 30 2026-02-20T12:00:35.675 INFO:tasks.rados.rados.0.trial153.stdout:308: oid 36 version 30 is already newer than 30 2026-02-20T12:00:35.675 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 36 v 30 (ObjNum 168 snap 37 seq_num 168) dirty exists 2026-02-20T12:00:35.675 INFO:tasks.rados.rados.0.trial153.stdout:308: left oid 36 (ObjNum 168 snap 37 seq_num 168) 2026-02-20T12:00:36.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~83s 2026-02-20T12:00:36.497 INFO:tasks.rados.rados.0.trial153.stdout:306: done (2 left) 2026-02-20T12:00:36.498 INFO:tasks.rados.rados.0.trial153.stdout:308: done (1 left) 2026-02-20T12:00:36.498 INFO:tasks.rados.rados.0.trial153.stdout:309: done (0 left) 2026-02-20T12:00:36.498 INFO:tasks.rados.rados.0.trial153.stdout:310: read oid 26 snap 37 2026-02-20T12:00:36.498 INFO:tasks.rados.rados.0.trial153.stdout:310: expect (ObjNum 147 snap 25 seq_num 147) 2026-02-20T12:00:36.498 INFO:tasks.rados.rados.0.trial153.stdout:311: write oid 23 current snap is 38 2026-02-20T12:00:36.498 INFO:tasks.rados.rados.0.trial153.stdout:311: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2026-02-20T12:00:36.501 INFO:tasks.rados.rados.0.trial153.stdout:311: writing trial15342161-23 from 440383 to 917230 tid 1 2026-02-20T12:00:36.505 INFO:tasks.rados.rados.0.trial153.stdout:311: writing trial15342161-23 from 1556398 to 1993781 tid 2 2026-02-20T12:00:36.508 INFO:tasks.rados.rados.0.trial153.stdout:311: writing trial15342161-23 from 2474721 to 2840383 tid 3 2026-02-20T12:00:36.508 INFO:tasks.rados.rados.0.trial153.stdout:312: read oid 1 snap -1 2026-02-20T12:00:36.508 INFO:tasks.rados.rados.0.trial153.stdout:312: expect (ObjNum 151 snap 26 seq_num 151) 2026-02-20T12:00:36.508 INFO:tasks.rados.rados.0.trial153.stdout:313: snap_create 2026-02-20T12:00:36.530 INFO:tasks.rados.rados.0.trial153.stdout:312: expect (ObjNum 151 snap 26 seq_num 151) 2026-02-20T12:00:36.602 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:36.638 INFO:tasks.rados.rados.0.trial153.stdout:310: expect (ObjNum 147 snap 25 seq_num 147) 2026-02-20T12:00:36.638 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:36.685 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:36.777 INFO:tasks.rados.rados.0.trial153.stdout:311: finishing write tid 1 to trial15342161-23 2026-02-20T12:00:36.778 INFO:tasks.rados.rados.0.trial153.stdout:311: finishing write tid 2 to trial15342161-23 2026-02-20T12:00:36.778 INFO:tasks.rados.rados.0.trial153.stdout:311: finishing write tid 3 to trial15342161-23 2026-02-20T12:00:36.778 INFO:tasks.rados.rados.0.trial153.stdout:311: finishing write tid 4 to trial15342161-23 2026-02-20T12:00:36.778 INFO:tasks.rados.rados.0.trial153.stdout:311: finishing write tid 5 to trial15342161-23 2026-02-20T12:00:36.778 INFO:tasks.rados.rados.0.trial153.stdout:311: oid 23 updating version 0 to 72 2026-02-20T12:00:36.778 INFO:tasks.rados.rados.0.trial153.stdout:311: oid 23 version 72 is already newer than 72 2026-02-20T12:00:36.778 INFO:tasks.rados.rados.0.trial153.stdout:311: oid 23 version 72 is already newer than 72 2026-02-20T12:00:36.778 INFO:tasks.rados.rados.0.trial153.stdout:311: oid 23 version 72 is already newer than 72 2026-02-20T12:00:36.778 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 23 v 72 (ObjNum 169 snap 38 seq_num 169) dirty exists 2026-02-20T12:00:36.778 INFO:tasks.rados.rados.0.trial153.stdout:311: left oid 23 (ObjNum 169 snap 38 seq_num 169) 2026-02-20T12:00:37.499 INFO:tasks.rados.rados.0.trial153.stdout:310: done (3 left) 2026-02-20T12:00:37.500 INFO:tasks.rados.rados.0.trial153.stdout:311: done (2 left) 2026-02-20T12:00:37.500 INFO:tasks.rados.rados.0.trial153.stdout:312: done (1 left) 2026-02-20T12:00:37.500 INFO:tasks.rados.rados.0.trial153.stdout:313: done (0 left) 2026-02-20T12:00:37.500 INFO:tasks.rados.rados.0.trial153.stdout:314: write oid 37 current snap is 39 2026-02-20T12:00:37.500 INFO:tasks.rados.rados.0.trial153.stdout:314: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2026-02-20T12:00:37.503 INFO:tasks.rados.rados.0.trial153.stdout:314: writing trial15342161-37 from 457190 to 1009286 tid 1 2026-02-20T12:00:37.508 INFO:tasks.rados.rados.0.trial153.stdout:314: writing trial15342161-37 from 1614880 to 2195921 tid 2 2026-02-20T12:00:37.511 INFO:tasks.rados.rados.0.trial153.stdout:314: writing trial15342161-37 from 2785791 to 2857190 tid 3 2026-02-20T12:00:37.511 INFO:tasks.rados.rados.0.trial153.stdout:315: write oid 9 current snap is 39 2026-02-20T12:00:37.511 INFO:tasks.rados.rados.0.trial153.stdout:315: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2026-02-20T12:00:37.514 INFO:tasks.rados.rados.0.trial153.stdout:315: writing trial15342161-9 from 473997 to 1101342 tid 1 2026-02-20T12:00:37.518 INFO:tasks.rados.rados.0.trial153.stdout:315: writing trial15342161-9 from 1673362 to 2114414 tid 2 2026-02-20T12:00:37.521 INFO:tasks.rados.rados.0.trial153.stdout:315: writing trial15342161-9 from 2529567 to 2873997 tid 3 2026-02-20T12:00:37.521 INFO:tasks.rados.rados.0.trial153.stdout:316: delete oid 24 current snap is 39 2026-02-20T12:00:37.558 INFO:tasks.rados.rados.0.trial153.stdout:316: done (2 left) 2026-02-20T12:00:37.558 INFO:tasks.rados.rados.0.trial153.stdout:317: snap_remove snap 37 2026-02-20T12:00:37.605 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:37.682 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:37.676+0000 7fb10f8c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:37.684 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:37.680+0000 7fb10f8c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:37.713 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:37.729 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:38.500 INFO:tasks.rados.rados.0.trial153.stdout:317: done (2 left) 2026-02-20T12:00:38.500 INFO:tasks.rados.rados.0.trial153.stdout:318: read oid 48 snap -1 2026-02-20T12:00:38.500 INFO:tasks.rados.rados.0.trial153.stdout:318: expect deleted 2026-02-20T12:00:38.500 INFO:tasks.rados.rados.0.trial153.stdout:319: write (excl) oid 32 current snap is 39 2026-02-20T12:00:38.500 INFO:tasks.rados.rados.0.trial153.stdout:315: finishing write tid 1 to trial15342161-9 2026-02-20T12:00:38.500 INFO:tasks.rados.rados.0.trial153.stdout:319: seq_num 173 ranges {507611=777843,2073973=444721} 2026-02-20T12:00:38.505 INFO:tasks.rados.rados.0.trial153.stdout:319: writing trial15342161-32 from 507611 to 1285454 tid 1 2026-02-20T12:00:38.510 INFO:tasks.rados.rados.0.trial153.stdout:319: writing trial15342161-32 from 2073973 to 2518694 tid 2 2026-02-20T12:00:38.510 INFO:tasks.rados.rados.0.trial153.stdout:320: read oid 21 snap -1 2026-02-20T12:00:38.510 INFO:tasks.rados.rados.0.trial153.stdout:320: expect deleted 2026-02-20T12:00:38.510 INFO:tasks.rados.rados.0.trial153.stdout:321: read oid 42 snap -1 2026-02-20T12:00:38.510 INFO:tasks.rados.rados.0.trial153.stdout:321: expect deleted 2026-02-20T12:00:38.510 INFO:tasks.rados.rados.0.trial153.stdout:322: delete oid 5 current snap is 39 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:315: finishing write tid 2 to trial15342161-9 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:315: finishing write tid 3 to trial15342161-9 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:315: finishing write tid 4 to trial15342161-9 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:315: finishing write tid 5 to trial15342161-9 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:315: oid 9 updating version 0 to 99 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:315: oid 9 version 99 is already newer than 99 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:315: oid 9 version 99 is already newer than 99 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:315: oid 9 version 99 is already newer than 99 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 9 v 99 (ObjNum 171 snap 39 seq_num 171) dirty exists 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:315: left oid 9 (ObjNum 171 snap 39 seq_num 171) 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:314: finishing write tid 1 to trial15342161-37 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:314: finishing write tid 2 to trial15342161-37 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:314: finishing write tid 3 to trial15342161-37 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:314: finishing write tid 4 to trial15342161-37 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:314: finishing write tid 5 to trial15342161-37 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:314: oid 37 updating version 0 to 37 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:314: oid 37 version 37 is already newer than 37 2026-02-20T12:00:38.511 INFO:tasks.rados.rados.0.trial153.stdout:314: oid 37 version 37 is already newer than 37 2026-02-20T12:00:38.512 INFO:tasks.rados.rados.0.trial153.stdout:314: oid 37 version 37 is already newer than 37 2026-02-20T12:00:38.512 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 37 (ObjNum 170 snap 39 seq_num 170) dirty exists 2026-02-20T12:00:38.512 INFO:tasks.rados.rados.0.trial153.stdout:314: left oid 37 (ObjNum 170 snap 39 seq_num 170) 2026-02-20T12:00:38.542 INFO:tasks.rados.rados.0.trial153.stdout:314: done (6 left) 2026-02-20T12:00:38.542 INFO:tasks.rados.rados.0.trial153.stdout:315: done (5 left) 2026-02-20T12:00:38.542 INFO:tasks.rados.rados.0.trial153.stdout:318: done (4 left) 2026-02-20T12:00:38.542 INFO:tasks.rados.rados.0.trial153.stdout:320: done (3 left) 2026-02-20T12:00:38.542 INFO:tasks.rados.rados.0.trial153.stdout:321: done (2 left) 2026-02-20T12:00:38.542 INFO:tasks.rados.rados.0.trial153.stdout:322: done (1 left) 2026-02-20T12:00:38.542 INFO:tasks.rados.rados.0.trial153.stdout:323: write oid 37 current snap is 39 2026-02-20T12:00:38.543 INFO:tasks.rados.rados.0.trial153.stdout:323: seq_num 175 ranges {541225=644694,1907290=448390} 2026-02-20T12:00:38.547 INFO:tasks.rados.rados.0.trial153.stdout:323: writing trial15342161-37 from 541225 to 1185919 tid 1 2026-02-20T12:00:38.551 INFO:tasks.rados.rados.0.trial153.stdout:323: writing trial15342161-37 from 1907290 to 2355680 tid 2 2026-02-20T12:00:38.552 INFO:tasks.rados.rados.0.trial153.stdout:324: write (excl) oid 49 current snap is 39 2026-02-20T12:00:38.552 INFO:tasks.rados.rados.0.trial153.stdout:324: seq_num 176 ranges {558032=719943,1849419=592048} 2026-02-20T12:00:38.556 INFO:tasks.rados.rados.0.trial153.stdout:324: writing trial15342161-49 from 558032 to 1277975 tid 1 2026-02-20T12:00:38.561 INFO:tasks.rados.rados.0.trial153.stdout:324: writing trial15342161-49 from 1849419 to 2441467 tid 2 2026-02-20T12:00:38.561 INFO:tasks.rados.rados.0.trial153.stdout:325: read oid 24 snap -1 2026-02-20T12:00:38.561 INFO:tasks.rados.rados.0.trial153.stdout:325: expect deleted 2026-02-20T12:00:38.561 INFO:tasks.rados.rados.0.trial153.stdout:326: snap_remove snap 27 2026-02-20T12:00:38.606 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:38.677 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:38.680 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:39.506 INFO:tasks.rados.rados.0.trial153.stdout:326: done (4 left) 2026-02-20T12:00:39.506 INFO:tasks.rados.rados.0.trial153.stdout:327: snap_remove snap 36 2026-02-20T12:00:39.729 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:39.805 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:39.800+0000 7fea20a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:39.808 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:39.803+0000 7fea20a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:39.837 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:39.853 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:40.509 INFO:tasks.rados.rados.0.trial153.stdout:327: done (4 left) 2026-02-20T12:00:40.509 INFO:tasks.rados.rados.0.trial153.stdout:328: read oid 6 snap -1 2026-02-20T12:00:40.509 INFO:tasks.rados.rados.0.trial153.stdout:328: expect deleted 2026-02-20T12:00:40.510 INFO:tasks.rados.rados.0.trial153.stdout:329: write oid 3 current snap is 39 2026-02-20T12:00:40.510 INFO:tasks.rados.rados.0.trial153.stdout:319: finishing write tid 1 to trial15342161-32 2026-02-20T12:00:40.510 INFO:tasks.rados.rados.0.trial153.stdout:329: seq_num 177 ranges {574839=795192,1907901=452059} 2026-02-20T12:00:40.514 INFO:tasks.rados.rados.0.trial153.stdout:329: writing trial15342161-3 from 574839 to 1370031 tid 1 2026-02-20T12:00:40.520 INFO:tasks.rados.rados.0.trial153.stdout:329: writing trial15342161-3 from 1907901 to 2359960 tid 2 2026-02-20T12:00:40.520 INFO:tasks.rados.rados.0.trial153.stdout:330: delete oid 7 current snap is 39 2026-02-20T12:00:40.520 INFO:tasks.rados.rados.0.trial153.stdout:319: finishing write tid 2 to trial15342161-32 2026-02-20T12:00:40.520 INFO:tasks.rados.rados.0.trial153.stdout:323: finishing write tid 1 to trial15342161-37 2026-02-20T12:00:40.521 INFO:tasks.rados.rados.0.trial153.stdout:319: finishing write tid 3 to trial15342161-32 2026-02-20T12:00:40.521 INFO:tasks.rados.rados.0.trial153.stdout:319: finishing write tid 4 to trial15342161-32 2026-02-20T12:00:40.521 INFO:tasks.rados.rados.0.trial153.stdout:319: oid 32 updating version 0 to 64 2026-02-20T12:00:40.521 INFO:tasks.rados.rados.0.trial153.stdout:319: oid 32 version 64 is already newer than 62 2026-02-20T12:00:40.521 INFO:tasks.rados.rados.0.trial153.stdout:319: oid 32 version 64 is already newer than 64 2026-02-20T12:00:40.521 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 32 v 64 (ObjNum 173 snap 39 seq_num 173) dirty exists 2026-02-20T12:00:40.521 INFO:tasks.rados.rados.0.trial153.stdout:319: left oid 32 (ObjNum 173 snap 39 seq_num 173) 2026-02-20T12:00:40.521 INFO:tasks.rados.rados.0.trial153.stdout:323: finishing write tid 2 to trial15342161-37 2026-02-20T12:00:40.521 INFO:tasks.rados.rados.0.trial153.stdout:323: finishing write tid 3 to trial15342161-37 2026-02-20T12:00:40.521 INFO:tasks.rados.rados.0.trial153.stdout:323: finishing write tid 4 to trial15342161-37 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:323: oid 37 updating version 0 to 42 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:323: oid 37 version 42 is already newer than 42 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:323: oid 37 version 42 is already newer than 42 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 42 (ObjNum 175 snap 39 seq_num 175) dirty exists 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:323: left oid 37 (ObjNum 175 snap 39 seq_num 175) 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:324: finishing write tid 1 to trial15342161-49 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:324: finishing write tid 2 to trial15342161-49 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:324: finishing write tid 3 to trial15342161-49 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:324: finishing write tid 4 to trial15342161-49 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:324: oid 49 updating version 0 to 54 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:324: oid 49 version 54 is already newer than 54 2026-02-20T12:00:40.522 INFO:tasks.rados.rados.0.trial153.stdout:324: oid 49 version 54 is already newer than 52 2026-02-20T12:00:40.523 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 49 v 54 (ObjNum 176 snap 39 seq_num 176) dirty exists 2026-02-20T12:00:40.523 INFO:tasks.rados.rados.0.trial153.stdout:324: left oid 49 (ObjNum 176 snap 39 seq_num 176) 2026-02-20T12:00:40.556 INFO:tasks.rados.rados.0.trial153.stdout:329: finishing write tid 1 to trial15342161-3 2026-02-20T12:00:40.558 INFO:tasks.rados.rados.0.trial153.stdout:319: done (6 left) 2026-02-20T12:00:40.558 INFO:tasks.rados.rados.0.trial153.stdout:323: done (5 left) 2026-02-20T12:00:40.558 INFO:tasks.rados.rados.0.trial153.stdout:324: done (4 left) 2026-02-20T12:00:40.558 INFO:tasks.rados.rados.0.trial153.stdout:325: done (3 left) 2026-02-20T12:00:40.558 INFO:tasks.rados.rados.0.trial153.stdout:328: done (2 left) 2026-02-20T12:00:40.558 INFO:tasks.rados.rados.0.trial153.stdout:330: done (1 left) 2026-02-20T12:00:40.558 INFO:tasks.rados.rados.0.trial153.stdout:331: read oid 9 snap -1 2026-02-20T12:00:40.558 INFO:tasks.rados.rados.0.trial153.stdout:331: expect (ObjNum 171 snap 39 seq_num 171) 2026-02-20T12:00:40.559 INFO:tasks.rados.rados.0.trial153.stdout:332: snap_remove snap 35 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:332: done (2 left) 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:333: snap_create 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:329: finishing write tid 2 to trial15342161-3 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:329: finishing write tid 3 to trial15342161-3 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:329: finishing write tid 4 to trial15342161-3 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:329: oid 3 updating version 0 to 68 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:329: oid 3 version 68 is already newer than 68 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:329: oid 3 version 68 is already newer than 68 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 3 v 68 (ObjNum 177 snap 39 seq_num 177) dirty exists 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:329: left oid 3 (ObjNum 177 snap 39 seq_num 177) 2026-02-20T12:00:41.514 INFO:tasks.rados.rados.0.trial153.stdout:331: expect (ObjNum 171 snap 39 seq_num 171) 2026-02-20T12:00:41.597 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1243585 2026-02-20T12:00:41.853 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:41.870 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~88s 2026-02-20T12:00:41.932 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:41.926+0000 7fee26ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:41.934 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:41.929+0000 7fee26ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:41.961 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:41.977 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:42.516 INFO:tasks.rados.rados.0.trial153.stdout:329: done (2 left) 2026-02-20T12:00:42.516 INFO:tasks.rados.rados.0.trial153.stdout:331: done (1 left) 2026-02-20T12:00:42.516 INFO:tasks.rados.rados.0.trial153.stdout:333: done (0 left) 2026-02-20T12:00:42.516 INFO:tasks.rados.rados.0.trial153.stdout:334: snap_create 2026-02-20T12:00:43.523 INFO:tasks.rados.rados.0.trial153.stdout:334: done (0 left) 2026-02-20T12:00:43.523 INFO:tasks.rados.rados.0.trial153.stdout:335: write (excl) oid 14 current snap is 41 2026-02-20T12:00:43.523 INFO:tasks.rados.rados.0.trial153.stdout:335: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2026-02-20T12:00:43.527 INFO:tasks.rados.rados.0.trial153.stdout:335: writing trial15342161-14 from 608453 to 1154143 tid 1 2026-02-20T12:00:43.531 INFO:tasks.rados.rados.0.trial153.stdout:335: writing trial15342161-14 from 1624865 to 2080593 tid 2 2026-02-20T12:00:43.535 INFO:tasks.rados.rados.0.trial153.stdout:335: writing trial15342161-14 from 2632598 to 3008453 tid 3 2026-02-20T12:00:43.536 INFO:tasks.rados.rados.0.trial153.stdout:336: write (excl) oid 17 current snap is 41 2026-02-20T12:00:43.536 INFO:tasks.rados.rados.0.trial153.stdout:336: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2026-02-20T12:00:43.540 INFO:tasks.rados.rados.0.trial153.stdout:336: writing trial15342161-17 from 625260 to 1246199 tid 1 2026-02-20T12:00:43.544 INFO:tasks.rados.rados.0.trial153.stdout:336: writing trial15342161-17 from 1683347 to 2282733 tid 2 2026-02-20T12:00:43.547 INFO:tasks.rados.rados.0.trial153.stdout:336: writing trial15342161-17 from 2943668 to 3025260 tid 3 2026-02-20T12:00:43.548 INFO:tasks.rados.rados.0.trial153.stdout:337: write oid 24 current snap is 41 2026-02-20T12:00:43.548 INFO:tasks.rados.rados.0.trial153.stdout:337: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2026-02-20T12:00:43.553 INFO:tasks.rados.rados.0.trial153.stdout:337: writing trial15342161-24 from 642067 to 1338255 tid 1 2026-02-20T12:00:43.557 INFO:tasks.rados.rados.0.trial153.stdout:337: writing trial15342161-24 from 2025476 to 2484873 tid 2 2026-02-20T12:00:43.559 INFO:tasks.rados.rados.0.trial153.stdout:337: writing trial15342161-24 from 2971091 to 3042067 tid 3 2026-02-20T12:00:43.560 INFO:tasks.rados.rados.0.trial153.stdout:338: snap_remove snap 31 2026-02-20T12:00:43.681 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:43.753 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:43.756 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:43.978 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:44.057 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:44.051+0000 7f1da988b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:44.059 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:44.054+0000 7f1da988b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:44.086 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:44.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:44.526 INFO:tasks.rados.rados.0.trial153.stdout:338: done (3 left) 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:339: snap_create 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:335: finishing write tid 1 to trial15342161-14 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:336: finishing write tid 1 to trial15342161-17 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:337: finishing write tid 1 to trial15342161-24 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:335: finishing write tid 2 to trial15342161-14 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:335: finishing write tid 3 to trial15342161-14 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:337: finishing write tid 2 to trial15342161-24 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:335: finishing write tid 4 to trial15342161-14 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:337: finishing write tid 3 to trial15342161-24 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:336: finishing write tid 2 to trial15342161-17 2026-02-20T12:00:44.527 INFO:tasks.rados.rados.0.trial153.stdout:335: finishing write tid 5 to trial15342161-14 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:335: oid 14 updating version 0 to 55 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:335: oid 14 version 55 is already newer than 55 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:335: oid 14 version 55 is already newer than 55 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:335: oid 14 version 55 is already newer than 52 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 55 (ObjNum 179 snap 41 seq_num 179) dirty exists 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:335: left oid 14 (ObjNum 179 snap 41 seq_num 179) 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:337: finishing write tid 4 to trial15342161-24 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:336: finishing write tid 3 to trial15342161-17 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:337: finishing write tid 5 to trial15342161-24 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:337: oid 24 updating version 0 to 19 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:337: oid 24 version 19 is already newer than 19 2026-02-20T12:00:44.528 INFO:tasks.rados.rados.0.trial153.stdout:337: oid 24 version 19 is already newer than 19 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:337: oid 24 version 19 is already newer than 19 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 24 v 19 (ObjNum 181 snap 41 seq_num 181) dirty exists 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:337: left oid 24 (ObjNum 181 snap 41 seq_num 181) 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:336: finishing write tid 4 to trial15342161-17 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:336: finishing write tid 5 to trial15342161-17 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:336: oid 17 updating version 0 to 63 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:336: oid 17 version 63 is already newer than 63 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:336: oid 17 version 63 is already newer than 60 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:336: oid 17 version 63 is already newer than 63 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 17 v 63 (ObjNum 180 snap 41 seq_num 180) dirty exists 2026-02-20T12:00:44.529 INFO:tasks.rados.rados.0.trial153.stdout:336: left oid 17 (ObjNum 180 snap 41 seq_num 180) 2026-02-20T12:00:45.530 INFO:tasks.rados.rados.0.trial153.stdout:335: done (3 left) 2026-02-20T12:00:45.530 INFO:tasks.rados.rados.0.trial153.stdout:336: done (2 left) 2026-02-20T12:00:45.531 INFO:tasks.rados.rados.0.trial153.stdout:337: done (1 left) 2026-02-20T12:00:45.531 INFO:tasks.rados.rados.0.trial153.stdout:339: done (0 left) 2026-02-20T12:00:45.531 INFO:tasks.rados.rados.0.trial153.stdout:340: delete oid 12 current snap is 42 2026-02-20T12:00:45.568 INFO:tasks.rados.rados.0.trial153.stdout:340: done (0 left) 2026-02-20T12:00:45.569 INFO:tasks.rados.rados.0.trial153.stdout:341: write (excl) oid 43 current snap is 42 2026-02-20T12:00:45.569 INFO:tasks.rados.rados.0.trial153.stdout:341: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2026-02-20T12:00:45.573 INFO:tasks.rados.rados.0.trial153.stdout:341: writing trial15342161-43 from 675681 to 1238720 tid 1 2026-02-20T12:00:45.578 INFO:tasks.rados.rados.0.trial153.stdout:341: writing trial15342161-43 from 1858793 to 2605506 tid 2 2026-02-20T12:00:45.580 INFO:tasks.rados.rados.0.trial153.stdout:341: writing trial15342161-43 from 3025937 to 3075681 tid 3 2026-02-20T12:00:45.580 INFO:tasks.rados.rados.0.trial153.stdout:342: snap_create 2026-02-20T12:00:45.605 INFO:tasks.rados.rados.0.trial153.stdout:341: finishing write tid 1 to trial15342161-43 2026-02-20T12:00:45.619 INFO:tasks.rados.rados.0.trial153.stdout:341: finishing write tid 2 to trial15342161-43 2026-02-20T12:00:45.625 INFO:tasks.rados.rados.0.trial153.stdout:341: finishing write tid 3 to trial15342161-43 2026-02-20T12:00:45.627 INFO:tasks.rados.rados.0.trial153.stdout:341: finishing write tid 4 to trial15342161-43 2026-02-20T12:00:45.629 INFO:tasks.rados.rados.0.trial153.stdout:341: finishing write tid 5 to trial15342161-43 2026-02-20T12:00:45.630 INFO:tasks.rados.rados.0.trial153.stdout:341: oid 43 updating version 0 to 32 2026-02-20T12:00:45.630 INFO:tasks.rados.rados.0.trial153.stdout:341: oid 43 version 32 is already newer than 32 2026-02-20T12:00:45.630 INFO:tasks.rados.rados.0.trial153.stdout:341: oid 43 version 32 is already newer than 32 2026-02-20T12:00:45.630 INFO:tasks.rados.rados.0.trial153.stdout:341: oid 43 version 32 is already newer than 29 2026-02-20T12:00:45.630 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 43 v 32 (ObjNum 183 snap 42 seq_num 183) dirty exists 2026-02-20T12:00:45.630 INFO:tasks.rados.rados.0.trial153.stdout:341: left oid 43 (ObjNum 183 snap 42 seq_num 183) 2026-02-20T12:00:46.101 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:46.178 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:46.172+0000 7f98582b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:46.181 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:46.176+0000 7f98582b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:46.207 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:46.220 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:46.502 INFO:tasks.rados.rados.0.trial153.stdout:341: done (1 left) 2026-02-20T12:00:46.503 INFO:tasks.rados.rados.0.trial153.stdout:342: done (0 left) 2026-02-20T12:00:46.503 INFO:tasks.rados.rados.0.trial153.stdout:343: snap_create 2026-02-20T12:00:47.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~94s 2026-02-20T12:00:47.504 INFO:tasks.rados.rados.0.trial153.stdout:343: done (0 left) 2026-02-20T12:00:47.504 INFO:tasks.rados.rados.0.trial153.stdout:344: write oid 45 current snap is 44 2026-02-20T12:00:47.504 INFO:tasks.rados.rados.0.trial153.stdout:344: seq_num 184 ranges {692488=638288,1917275=606724} 2026-02-20T12:00:47.509 INFO:tasks.rados.rados.0.trial153.stdout:344: writing trial15342161-45 from 692488 to 1330776 tid 1 2026-02-20T12:00:47.514 INFO:tasks.rados.rados.0.trial153.stdout:344: writing trial15342161-45 from 1917275 to 2523999 tid 2 2026-02-20T12:00:47.514 INFO:tasks.rados.rados.0.trial153.stdout:345: snap_create 2026-02-20T12:00:47.549 INFO:tasks.rados.rados.0.trial153.stdout:344: finishing write tid 1 to trial15342161-45 2026-02-20T12:00:47.555 INFO:tasks.rados.rados.0.trial153.stdout:344: finishing write tid 2 to trial15342161-45 2026-02-20T12:00:47.559 INFO:tasks.rados.rados.0.trial153.stdout:344: finishing write tid 3 to trial15342161-45 2026-02-20T12:00:47.560 INFO:tasks.rados.rados.0.trial153.stdout:344: finishing write tid 4 to trial15342161-45 2026-02-20T12:00:47.561 INFO:tasks.rados.rados.0.trial153.stdout:344: oid 45 updating version 0 to 36 2026-02-20T12:00:47.561 INFO:tasks.rados.rados.0.trial153.stdout:344: oid 45 version 36 is already newer than 36 2026-02-20T12:00:47.561 INFO:tasks.rados.rados.0.trial153.stdout:344: oid 45 version 36 is already newer than 36 2026-02-20T12:00:47.561 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 36 (ObjNum 184 snap 44 seq_num 184) dirty exists 2026-02-20T12:00:47.561 INFO:tasks.rados.rados.0.trial153.stdout:344: left oid 45 (ObjNum 184 snap 44 seq_num 184) 2026-02-20T12:00:48.222 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:48.301 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:48.295+0000 7fecacbdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:48.303 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:48.298+0000 7fecacbdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:48.333 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:48.349 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:48.505 INFO:tasks.rados.rados.0.trial153.stdout:344: done (1 left) 2026-02-20T12:00:48.506 INFO:tasks.rados.rados.0.trial153.stdout:345: done (0 left) 2026-02-20T12:00:48.506 INFO:tasks.rados.rados.0.trial153.stdout:346: snap_create 2026-02-20T12:00:48.757 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:48.830 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:48.833 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:49.510 INFO:tasks.rados.rados.0.trial153.stdout:346: done (0 left) 2026-02-20T12:00:49.510 INFO:tasks.rados.rados.0.trial153.stdout:347: delete oid 13 current snap is 46 2026-02-20T12:00:49.555 INFO:tasks.rados.rados.0.trial153.stdout:347: done (0 left) 2026-02-20T12:00:49.555 INFO:tasks.rados.rados.0.trial153.stdout:348: write (excl) oid 36 current snap is 46 2026-02-20T12:00:49.555 INFO:tasks.rados.rados.0.trial153.stdout:348: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2026-02-20T12:00:49.560 INFO:tasks.rados.rados.0.trial153.stdout:348: writing trial15342161-36 from 726102 to 1514888 tid 1 2026-02-20T12:00:49.564 INFO:tasks.rados.rados.0.trial153.stdout:348: writing trial15342161-36 from 1917886 to 2528279 tid 2 2026-02-20T12:00:49.566 INFO:tasks.rados.rados.0.trial153.stdout:348: writing trial15342161-36 from 3108206 to 3126102 tid 3 2026-02-20T12:00:49.567 INFO:tasks.rados.rados.0.trial153.stdout:349: snap_create 2026-02-20T12:00:49.589 INFO:tasks.rados.rados.0.trial153.stdout:348: finishing write tid 1 to trial15342161-36 2026-02-20T12:00:49.603 INFO:tasks.rados.rados.0.trial153.stdout:348: finishing write tid 2 to trial15342161-36 2026-02-20T12:00:49.608 INFO:tasks.rados.rados.0.trial153.stdout:348: finishing write tid 3 to trial15342161-36 2026-02-20T12:00:49.611 INFO:tasks.rados.rados.0.trial153.stdout:348: finishing write tid 4 to trial15342161-36 2026-02-20T12:00:49.613 INFO:tasks.rados.rados.0.trial153.stdout:348: finishing write tid 5 to trial15342161-36 2026-02-20T12:00:49.613 INFO:tasks.rados.rados.0.trial153.stdout:348: oid 36 updating version 0 to 37 2026-02-20T12:00:49.613 INFO:tasks.rados.rados.0.trial153.stdout:348: oid 36 version 37 is already newer than 37 2026-02-20T12:00:49.613 INFO:tasks.rados.rados.0.trial153.stdout:348: oid 36 version 37 is already newer than 34 2026-02-20T12:00:49.613 INFO:tasks.rados.rados.0.trial153.stdout:348: oid 36 version 37 is already newer than 37 2026-02-20T12:00:49.613 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 36 v 37 (ObjNum 186 snap 46 seq_num 186) dirty exists 2026-02-20T12:00:49.613 INFO:tasks.rados.rados.0.trial153.stdout:348: left oid 36 (ObjNum 186 snap 46 seq_num 186) 2026-02-20T12:00:50.350 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:50.426 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:50.420+0000 7f9196e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:50.428 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:50.424+0000 7f9196e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:50.454 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:50.468 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:50.514 INFO:tasks.rados.rados.0.trial153.stdout:348: done (1 left) 2026-02-20T12:00:50.514 INFO:tasks.rados.rados.0.trial153.stdout:349: done (0 left) 2026-02-20T12:00:50.514 INFO:tasks.rados.rados.0.trial153.stdout:350: read oid 48 snap -1 2026-02-20T12:00:50.515 INFO:tasks.rados.rados.0.trial153.stdout:350: expect deleted 2026-02-20T12:00:50.515 INFO:tasks.rados.rados.0.trial153.stdout:351: write (excl) oid 36 current snap is 47 2026-02-20T12:00:50.515 INFO:tasks.rados.rados.0.trial153.stdout:351: seq_num 187 ranges {742909=464035,1976368=754051} 2026-02-20T12:00:50.518 INFO:tasks.rados.rados.0.trial153.stdout:351: writing trial15342161-36 from 742909 to 1206944 tid 1 2026-02-20T12:00:50.524 INFO:tasks.rados.rados.0.trial153.stdout:351: writing trial15342161-36 from 1976368 to 2730419 tid 2 2026-02-20T12:00:50.525 INFO:tasks.rados.rados.0.trial153.stdout:352: snap_remove snap 40 2026-02-20T12:00:51.516 INFO:tasks.rados.rados.0.trial153.stdout:352: done (2 left) 2026-02-20T12:00:51.516 INFO:tasks.rados.rados.0.trial153.stdout:353: write oid 50 current snap is 47 2026-02-20T12:00:51.517 INFO:tasks.rados.rados.0.trial153.stdout:353: seq_num 188 ranges {759716=539284,2034850=614062} 2026-02-20T12:00:51.521 INFO:tasks.rados.rados.0.trial153.stdout:353: writing trial15342161-50 from 759716 to 1299000 tid 1 2026-02-20T12:00:51.527 INFO:tasks.rados.rados.0.trial153.stdout:353: writing trial15342161-50 from 2034850 to 2648912 tid 2 2026-02-20T12:00:51.527 INFO:tasks.rados.rados.0.trial153.stdout:354: read oid 1 snap -1 2026-02-20T12:00:51.527 INFO:tasks.rados.rados.0.trial153.stdout:354: expect (ObjNum 151 snap 26 seq_num 151) 2026-02-20T12:00:51.527 INFO:tasks.rados.rados.0.trial153.stdout:355: read oid 45 snap 46 2026-02-20T12:00:51.527 INFO:tasks.rados.rados.0.trial153.stdout:355: expect (ObjNum 184 snap 44 seq_num 184) 2026-02-20T12:00:51.527 INFO:tasks.rados.rados.0.trial153.stdout:356: snap_remove snap 45 2026-02-20T12:00:52.469 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:52.519 INFO:tasks.rados.rados.0.trial153.stdout:356: done (5 left) 2026-02-20T12:00:52.520 INFO:tasks.rados.rados.0.trial153.stdout:357: delete oid 6 current snap is 47 2026-02-20T12:00:52.520 INFO:tasks.rados.rados.0.trial153.stdout:351: finishing write tid 1 to trial15342161-36 2026-02-20T12:00:52.520 INFO:tasks.rados.rados.0.trial153.stdout:351: finishing write tid 2 to trial15342161-36 2026-02-20T12:00:52.520 INFO:tasks.rados.rados.0.trial153.stdout:351: finishing write tid 3 to trial15342161-36 2026-02-20T12:00:52.520 INFO:tasks.rados.rados.0.trial153.stdout:351: finishing write tid 4 to trial15342161-36 2026-02-20T12:00:52.520 INFO:tasks.rados.rados.0.trial153.stdout:351: oid 36 updating version 0 to 41 2026-02-20T12:00:52.520 INFO:tasks.rados.rados.0.trial153.stdout:351: oid 36 version 41 is already newer than 41 2026-02-20T12:00:52.520 INFO:tasks.rados.rados.0.trial153.stdout:351: oid 36 version 41 is already newer than 39 2026-02-20T12:00:52.520 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 36 v 41 (ObjNum 187 snap 47 seq_num 187) dirty exists 2026-02-20T12:00:52.521 INFO:tasks.rados.rados.0.trial153.stdout:351: left oid 36 (ObjNum 187 snap 47 seq_num 187) 2026-02-20T12:00:52.521 INFO:tasks.rados.rados.0.trial153.stdout:355: expect (ObjNum 184 snap 44 seq_num 184) 2026-02-20T12:00:52.550 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:52.544+0000 7f8306e11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:52.552 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:52.547+0000 7f8306e11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:52.585 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:52.592 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 106261 2026-02-20T12:00:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:52.600 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:52.635 INFO:tasks.rados.rados.0.trial153.stdout:353: finishing write tid 1 to trial15342161-50 2026-02-20T12:00:52.635 INFO:tasks.rados.rados.0.trial153.stdout:354: expect (ObjNum 151 snap 26 seq_num 151) 2026-02-20T12:00:52.635 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:52.709 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1074258 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:353: finishing write tid 2 to trial15342161-50 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:350: done (5 left) 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:351: done (4 left) 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:354: done (3 left) 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:355: done (2 left) 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:357: done (1 left) 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:358: read oid 44 snap 44 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:358: expect (ObjNum 66 snap 3 seq_num 66) 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:359: write (excl) oid 9 current snap is 47 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:353: finishing write tid 3 to trial15342161-50 2026-02-20T12:00:52.735 INFO:tasks.rados.rados.0.trial153.stdout:353: finishing write tid 4 to trial15342161-50 2026-02-20T12:00:52.736 INFO:tasks.rados.rados.0.trial153.stdout:353: oid 50 updating version 0 to 61 2026-02-20T12:00:52.736 INFO:tasks.rados.rados.0.trial153.stdout:353: oid 50 version 61 is already newer than 61 2026-02-20T12:00:52.736 INFO:tasks.rados.rados.0.trial153.stdout:353: oid 50 version 61 is already newer than 61 2026-02-20T12:00:52.736 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 50 v 61 (ObjNum 188 snap 47 seq_num 188) dirty exists 2026-02-20T12:00:52.736 INFO:tasks.rados.rados.0.trial153.stdout:353: left oid 50 (ObjNum 188 snap 47 seq_num 188) 2026-02-20T12:00:52.736 INFO:tasks.rados.rados.0.trial153.stdout:359: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2026-02-20T12:00:52.741 INFO:tasks.rados.rados.0.trial153.stdout:359: writing trial15342161-9 from 793330 to 1483112 tid 1 2026-02-20T12:00:52.746 INFO:tasks.rados.rados.0.trial153.stdout:359: writing trial15342161-9 from 2035461 to 2653192 tid 2 2026-02-20T12:00:52.748 INFO:tasks.rados.rados.0.trial153.stdout:359: writing trial15342161-9 from 3101545 to 3193330 tid 3 2026-02-20T12:00:52.748 INFO:tasks.rados.rados.0.trial153.stdout:353: done (2 left) 2026-02-20T12:00:52.748 INFO:tasks.rados.rados.0.trial153.stdout:360: read oid 50 snap 28 2026-02-20T12:00:52.748 INFO:tasks.rados.rados.0.trial153.stdout:360: expect (ObjNum 49 snap 0 seq_num 49) 2026-02-20T12:00:52.748 INFO:tasks.rados.rados.0.trial153.stdout:361: read oid 16 snap 25 2026-02-20T12:00:52.748 INFO:tasks.rados.rados.0.trial153.stdout:361: expect (ObjNum 89 snap 10 seq_num 89) 2026-02-20T12:00:52.749 INFO:tasks.rados.rados.0.trial153.stdout:362: snap_remove snap 41 2026-02-20T12:00:52.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~99s 2026-02-20T12:00:53.522 INFO:tasks.rados.rados.0.trial153.stdout:362: done (4 left) 2026-02-20T12:00:53.522 INFO:tasks.rados.rados.0.trial153.stdout:363: snap_remove snap 34 2026-02-20T12:00:53.834 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:53.905 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:53.909 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:54.527 INFO:tasks.rados.rados.0.trial153.stdout:363: done (4 left) 2026-02-20T12:00:54.527 INFO:tasks.rados.rados.0.trial153.stdout:364: write oid 17 current snap is 47 2026-02-20T12:00:54.527 INFO:tasks.rados.rados.0.trial153.stdout:364: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2026-02-20T12:00:54.530 INFO:tasks.rados.rados.0.trial153.stdout:364: writing trial15342161-17 from 410137 to 891521 tid 1 2026-02-20T12:00:54.535 INFO:tasks.rados.rados.0.trial153.stdout:364: writing trial15342161-17 from 1410296 to 2171685 tid 2 2026-02-20T12:00:54.539 INFO:tasks.rados.rados.0.trial153.stdout:364: writing trial15342161-17 from 2728968 to 3210137 tid 3 2026-02-20T12:00:54.539 INFO:tasks.rados.rados.0.trial153.stdout:365: write oid 43 current snap is 47 2026-02-20T12:00:54.539 INFO:tasks.rados.rados.0.trial153.stdout:365: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2026-02-20T12:00:54.543 INFO:tasks.rados.rados.0.trial153.stdout:365: writing trial15342161-43 from 426944 to 983577 tid 1 2026-02-20T12:00:54.547 INFO:tasks.rados.rados.0.trial153.stdout:365: writing trial15342161-43 from 1468778 to 2090178 tid 2 2026-02-20T12:00:54.552 INFO:tasks.rados.rados.0.trial153.stdout:365: writing trial15342161-43 from 2872744 to 3226944 tid 3 2026-02-20T12:00:54.552 INFO:tasks.rados.rados.0.trial153.stdout:366: read oid 15 snap -1 2026-02-20T12:00:54.552 INFO:tasks.rados.rados.0.trial153.stdout:366: expect (ObjNum 161 snap 34 seq_num 161) 2026-02-20T12:00:54.552 INFO:tasks.rados.rados.0.trial153.stdout:367: delete oid 39 current snap is 47 2026-02-20T12:00:54.552 INFO:tasks.rados.rados.0.trial153.stdout:360: expect (ObjNum 49 snap 0 seq_num 49) 2026-02-20T12:00:54.561 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2477848 2026-02-20T12:00:54.564 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:54.588 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:54.601 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:54.612 INFO:tasks.rados.rados.0.trial153.stdout:358: expect (ObjNum 66 snap 3 seq_num 66) 2026-02-20T12:00:54.612 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:54.679 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:54.673+0000 7f0a60051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:54.682 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:54.677+0000 7f0a60051640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:54.686 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:54.709 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:54.722 INFO:tasks.rados.rados.0.trial153.stdout:359: finishing write tid 1 to trial15342161-9 2026-02-20T12:00:54.722 INFO:tasks.rados.rados.0.trial153.stdout:361: expect (ObjNum 89 snap 10 seq_num 89) 2026-02-20T12:00:54.725 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:54.725 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:54.834 INFO:tasks.rados.rados.0.trial153.stdout:359: finishing write tid 2 to trial15342161-9 2026-02-20T12:00:54.834 INFO:tasks.rados.rados.0.trial153.stdout:359: finishing write tid 3 to trial15342161-9 2026-02-20T12:00:54.834 INFO:tasks.rados.rados.0.trial153.stdout:358: done (7 left) 2026-02-20T12:00:54.834 INFO:tasks.rados.rados.0.trial153.stdout:360: done (6 left) 2026-02-20T12:00:54.834 INFO:tasks.rados.rados.0.trial153.stdout:361: done (5 left) 2026-02-20T12:00:54.834 INFO:tasks.rados.rados.0.trial153.stdout:367: done (4 left) 2026-02-20T12:00:54.834 INFO:tasks.rados.rados.0.trial153.stdout:368: snap_remove snap 42 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:368: done (4 left) 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:369: delete oid 47 current snap is 47 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:359: finishing write tid 4 to trial15342161-9 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:359: finishing write tid 5 to trial15342161-9 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:359: oid 9 updating version 0 to 118 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:359: oid 9 version 118 is already newer than 115 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:359: oid 9 version 118 is already newer than 118 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:359: oid 9 version 118 is already newer than 118 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 9 v 118 (ObjNum 190 snap 47 seq_num 190) dirty exists 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:359: left oid 9 (ObjNum 190 snap 47 seq_num 190) 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:364: finishing write tid 1 to trial15342161-17 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:364: finishing write tid 2 to trial15342161-17 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:365: finishing write tid 1 to trial15342161-43 2026-02-20T12:00:55.532 INFO:tasks.rados.rados.0.trial153.stdout:366: expect (ObjNum 161 snap 34 seq_num 161) 2026-02-20T12:00:55.539 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1302856 2026-02-20T12:00:55.622 INFO:tasks.rados.rados.0.trial153.stdout:365: finishing write tid 2 to trial15342161-43 2026-02-20T12:00:55.622 INFO:tasks.rados.rados.0.trial153.stdout:364: finishing write tid 3 to trial15342161-17 2026-02-20T12:00:55.622 INFO:tasks.rados.rados.0.trial153.stdout:365: finishing write tid 3 to trial15342161-43 2026-02-20T12:00:55.622 INFO:tasks.rados.rados.0.trial153.stdout:359: done (4 left) 2026-02-20T12:00:55.622 INFO:tasks.rados.rados.0.trial153.stdout:366: done (3 left) 2026-02-20T12:00:55.622 INFO:tasks.rados.rados.0.trial153.stdout:369: done (2 left) 2026-02-20T12:00:55.622 INFO:tasks.rados.rados.0.trial153.stdout:370: read oid 13 snap 25 2026-02-20T12:00:55.622 INFO:tasks.rados.rados.0.trial153.stdout:370: expect (ObjNum 122 snap 15 seq_num 122) 2026-02-20T12:00:55.622 INFO:tasks.rados.rados.0.trial153.stdout:371: write oid 47 current snap is 47 2026-02-20T12:00:55.623 INFO:tasks.rados.rados.0.trial153.stdout:364: finishing write tid 4 to trial15342161-17 2026-02-20T12:00:55.623 INFO:tasks.rados.rados.0.trial153.stdout:365: finishing write tid 4 to trial15342161-43 2026-02-20T12:00:55.623 INFO:tasks.rados.rados.0.trial153.stdout:371: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2026-02-20T12:00:55.627 INFO:tasks.rados.rados.0.trial153.stdout:371: writing trial15342161-47 from 477365 to 1259745 tid 1 2026-02-20T12:00:55.632 INFO:tasks.rados.rados.0.trial153.stdout:371: writing trial15342161-47 from 1927871 to 2696598 tid 2 2026-02-20T12:00:55.635 INFO:tasks.rados.rados.0.trial153.stdout:371: writing trial15342161-47 from 3122307 to 3277365 tid 3 2026-02-20T12:00:55.635 INFO:tasks.rados.rados.0.trial153.stdout:372: snap_remove snap 39 2026-02-20T12:00:56.501 INFO:tasks.rados.rados.0.trial153.stdout:372: done (4 left) 2026-02-20T12:00:56.501 INFO:tasks.rados.rados.0.trial153.stdout:373: read oid 8 snap -1 2026-02-20T12:00:56.501 INFO:tasks.rados.rados.0.trial153.stdout:373: expect (ObjNum 164 snap 34 seq_num 164) 2026-02-20T12:00:56.501 INFO:tasks.rados.rados.0.trial153.stdout:374: snap_remove snap 38 2026-02-20T12:00:56.726 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:00:56.801 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:56.795+0000 7f617e85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:56.803 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:56.799+0000 7f617e85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:56.830 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:56.846 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:57.503 INFO:tasks.rados.rados.0.trial153.stdout:374: done (5 left) 2026-02-20T12:00:57.503 INFO:tasks.rados.rados.0.trial153.stdout:375: read oid 42 snap -1 2026-02-20T12:00:57.504 INFO:tasks.rados.rados.0.trial153.stdout:375: expect deleted 2026-02-20T12:00:57.504 INFO:tasks.rados.rados.0.trial153.stdout:376: write (excl) oid 49 current snap is 47 2026-02-20T12:00:57.504 INFO:tasks.rados.rados.0.trial153.stdout:364: finishing write tid 5 to trial15342161-17 2026-02-20T12:00:57.504 INFO:tasks.rados.rados.0.trial153.stdout:364: oid 17 updating version 0 to 78 2026-02-20T12:00:57.504 INFO:tasks.rados.rados.0.trial153.stdout:364: oid 17 version 78 is already newer than 74 2026-02-20T12:00:57.504 INFO:tasks.rados.rados.0.trial153.stdout:364: oid 17 version 78 is already newer than 74 2026-02-20T12:00:57.504 INFO:tasks.rados.rados.0.trial153.stdout:364: oid 17 version 78 is already newer than 78 2026-02-20T12:00:57.504 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 17 v 78 (ObjNum 191 snap 47 seq_num 191) dirty exists 2026-02-20T12:00:57.504 INFO:tasks.rados.rados.0.trial153.stdout:364: left oid 17 (ObjNum 191 snap 47 seq_num 191) 2026-02-20T12:00:57.504 INFO:tasks.rados.rados.0.trial153.stdout:376: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2026-02-20T12:00:57.506 INFO:tasks.rados.rados.0.trial153.stdout:376: writing trial15342161-49 from 494172 to 951801 tid 1 2026-02-20T12:00:57.511 INFO:tasks.rados.rados.0.trial153.stdout:376: writing trial15342161-49 from 1586353 to 2098738 tid 2 2026-02-20T12:00:57.516 INFO:tasks.rados.rados.0.trial153.stdout:376: writing trial15342161-49 from 2749730 to 3294172 tid 3 2026-02-20T12:00:57.516 INFO:tasks.rados.rados.0.trial153.stdout:364: done (6 left) 2026-02-20T12:00:57.516 INFO:tasks.rados.rados.0.trial153.stdout:377: snap_remove snap 46 2026-02-20T12:00:58.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~105s 2026-02-20T12:00:58.503 INFO:tasks.rados.rados.0.trial153.stdout:377: done (6 left) 2026-02-20T12:00:58.504 INFO:tasks.rados.rados.0.trial153.stdout:378: snap_remove snap 44 2026-02-20T12:00:58.847 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:00:58.910 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:00:58.922 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:58.916+0000 7f95c5c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:58.924 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:00:58.920+0000 7f95c5c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:00:58.951 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:00:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:00:58.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:00:58.983 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:00:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:00:58.986 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:00:59.506 INFO:tasks.rados.rados.0.trial153.stdout:378: done (6 left) 2026-02-20T12:00:59.507 INFO:tasks.rados.rados.0.trial153.stdout:379: delete oid 1 current snap is 47 2026-02-20T12:00:59.507 INFO:tasks.rados.rados.0.trial153.stdout:365: finishing write tid 5 to trial15342161-43 2026-02-20T12:00:59.507 INFO:tasks.rados.rados.0.trial153.stdout:365: oid 43 updating version 0 to 41 2026-02-20T12:00:59.507 INFO:tasks.rados.rados.0.trial153.stdout:365: oid 43 version 41 is already newer than 41 2026-02-20T12:00:59.507 INFO:tasks.rados.rados.0.trial153.stdout:365: oid 43 version 41 is already newer than 41 2026-02-20T12:00:59.507 INFO:tasks.rados.rados.0.trial153.stdout:365: oid 43 version 41 is already newer than 41 2026-02-20T12:00:59.507 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 43 v 41 (ObjNum 192 snap 47 seq_num 192) dirty exists 2026-02-20T12:00:59.507 INFO:tasks.rados.rados.0.trial153.stdout:365: left oid 43 (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:00:59.507 INFO:tasks.rados.rados.0.trial153.stdout:370: expect (ObjNum 122 snap 15 seq_num 122) 2026-02-20T12:00:59.531 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 649917 2026-02-20T12:00:59.546 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 241151 2026-02-20T12:00:59.567 INFO:tasks.rados.rados.0.trial153.stdout:371: finishing write tid 1 to trial15342161-47 2026-02-20T12:00:59.567 INFO:tasks.rados.rados.0.trial153.stdout:371: finishing write tid 2 to trial15342161-47 2026-02-20T12:00:59.568 INFO:tasks.rados.rados.0.trial153.stdout:371: finishing write tid 3 to trial15342161-47 2026-02-20T12:00:59.568 INFO:tasks.rados.rados.0.trial153.stdout:371: finishing write tid 4 to trial15342161-47 2026-02-20T12:00:59.568 INFO:tasks.rados.rados.0.trial153.stdout:371: finishing write tid 5 to trial15342161-47 2026-02-20T12:00:59.568 INFO:tasks.rados.rados.0.trial153.stdout:371: oid 47 updating version 0 to 78 2026-02-20T12:00:59.568 INFO:tasks.rados.rados.0.trial153.stdout:371: oid 47 version 78 is already newer than 78 2026-02-20T12:00:59.568 INFO:tasks.rados.rados.0.trial153.stdout:371: oid 47 version 78 is already newer than 78 2026-02-20T12:00:59.568 INFO:tasks.rados.rados.0.trial153.stdout:371: oid 47 version 78 is already newer than 78 2026-02-20T12:00:59.568 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 47 v 78 (ObjNum 195 snap 47 seq_num 195) dirty exists 2026-02-20T12:00:59.568 INFO:tasks.rados.rados.0.trial153.stdout:371: left oid 47 (ObjNum 195 snap 47 seq_num 195) 2026-02-20T12:00:59.568 INFO:tasks.rados.rados.0.trial153.stdout:373: expect (ObjNum 164 snap 34 seq_num 164) 2026-02-20T12:00:59.598 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2269357 2026-02-20T12:00:59.626 INFO:tasks.rados.rados.0.trial153.stdout:376: finishing write tid 1 to trial15342161-49 2026-02-20T12:00:59.626 INFO:tasks.rados.rados.0.trial153.stdout:365: done (6 left) 2026-02-20T12:00:59.626 INFO:tasks.rados.rados.0.trial153.stdout:370: done (5 left) 2026-02-20T12:00:59.626 INFO:tasks.rados.rados.0.trial153.stdout:371: done (4 left) 2026-02-20T12:00:59.626 INFO:tasks.rados.rados.0.trial153.stdout:373: done (3 left) 2026-02-20T12:00:59.626 INFO:tasks.rados.rados.0.trial153.stdout:375: done (2 left) 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:379: done (1 left) 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:380: read oid 32 snap -1 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:380: expect (ObjNum 173 snap 39 seq_num 173) 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:381: write (excl) oid 37 current snap is 47 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:376: finishing write tid 2 to trial15342161-49 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:376: finishing write tid 3 to trial15342161-49 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:376: finishing write tid 4 to trial15342161-49 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:376: finishing write tid 5 to trial15342161-49 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:376: oid 49 updating version 0 to 89 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:376: oid 49 version 89 is already newer than 89 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:376: oid 49 version 89 is already newer than 86 2026-02-20T12:00:59.627 INFO:tasks.rados.rados.0.trial153.stdout:376: oid 49 version 89 is already newer than 89 2026-02-20T12:00:59.628 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 49 v 89 (ObjNum 196 snap 47 seq_num 196) dirty exists 2026-02-20T12:00:59.628 INFO:tasks.rados.rados.0.trial153.stdout:376: left oid 49 (ObjNum 196 snap 47 seq_num 196) 2026-02-20T12:00:59.628 INFO:tasks.rados.rados.0.trial153.stdout:381: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2026-02-20T12:00:59.630 INFO:tasks.rados.rados.0.trial153.stdout:381: writing trial15342161-37 from 527786 to 1252266 tid 1 2026-02-20T12:00:59.634 INFO:tasks.rados.rados.0.trial153.stdout:381: writing trial15342161-37 from 1703317 to 2219371 tid 2 2026-02-20T12:00:59.638 INFO:tasks.rados.rados.0.trial153.stdout:381: writing trial15342161-37 from 2804576 to 3239611 tid 3 2026-02-20T12:00:59.638 INFO:tasks.rados.rados.0.trial153.stdout:376: done (2 left) 2026-02-20T12:00:59.638 INFO:tasks.rados.rados.0.trial153.stdout:382: write oid 11 current snap is 47 2026-02-20T12:00:59.638 INFO:tasks.rados.rados.0.trial153.stdout:382: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2026-02-20T12:00:59.643 INFO:tasks.rados.rados.0.trial153.stdout:382: writing trial15342161-11 from 544593 to 1344322 tid 1 2026-02-20T12:00:59.647 INFO:tasks.rados.rados.0.trial153.stdout:382: writing trial15342161-11 from 1761799 to 2537864 tid 2 2026-02-20T12:00:59.650 INFO:tasks.rados.rados.0.trial153.stdout:382: writing trial15342161-11 from 2948352 to 3344593 tid 3 2026-02-20T12:00:59.651 INFO:tasks.rados.rados.0.trial153.stdout:383: snap_create 2026-02-20T12:00:59.652 INFO:tasks.rados.rados.0.trial153.stdout:380: expect (ObjNum 173 snap 39 seq_num 173) 2026-02-20T12:00:59.652 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:59.740 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:00:59.784 INFO:tasks.rados.rados.0.trial153.stdout:381: finishing write tid 1 to trial15342161-37 2026-02-20T12:00:59.785 INFO:tasks.rados.rados.0.trial153.stdout:382: finishing write tid 1 to trial15342161-11 2026-02-20T12:00:59.785 INFO:tasks.rados.rados.0.trial153.stdout:381: finishing write tid 2 to trial15342161-37 2026-02-20T12:00:59.785 INFO:tasks.rados.rados.0.trial153.stdout:382: finishing write tid 2 to trial15342161-11 2026-02-20T12:00:59.785 INFO:tasks.rados.rados.0.trial153.stdout:381: finishing write tid 3 to trial15342161-37 2026-02-20T12:00:59.785 INFO:tasks.rados.rados.0.trial153.stdout:382: finishing write tid 3 to trial15342161-11 2026-02-20T12:00:59.785 INFO:tasks.rados.rados.0.trial153.stdout:381: finishing write tid 4 to trial15342161-37 2026-02-20T12:00:59.785 INFO:tasks.rados.rados.0.trial153.stdout:382: finishing write tid 4 to trial15342161-11 2026-02-20T12:00:59.785 INFO:tasks.rados.rados.0.trial153.stdout:382: finishing write tid 5 to trial15342161-11 2026-02-20T12:00:59.785 INFO:tasks.rados.rados.0.trial153.stdout:382: oid 11 updating version 0 to 30 2026-02-20T12:00:59.785 INFO:tasks.rados.rados.0.trial153.stdout:382: oid 11 version 30 is already newer than 30 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:382: oid 11 version 30 is already newer than 30 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:382: oid 11 version 30 is already newer than 30 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 11 v 30 (ObjNum 199 snap 47 seq_num 199) dirty exists 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:382: left oid 11 (ObjNum 199 snap 47 seq_num 199) 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:381: finishing write tid 5 to trial15342161-37 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:381: oid 37 updating version 0 to 47 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:381: oid 37 version 47 is already newer than 47 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:381: oid 37 version 47 is already newer than 47 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:381: oid 37 version 47 is already newer than 44 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 47 (ObjNum 198 snap 47 seq_num 198) dirty exists 2026-02-20T12:00:59.786 INFO:tasks.rados.rados.0.trial153.stdout:381: left oid 37 (ObjNum 198 snap 47 seq_num 198) 2026-02-20T12:01:00.510 INFO:tasks.rados.rados.0.trial153.stdout:380: done (3 left) 2026-02-20T12:01:00.511 INFO:tasks.rados.rados.0.trial153.stdout:381: done (2 left) 2026-02-20T12:01:00.511 INFO:tasks.rados.rados.0.trial153.stdout:382: done (1 left) 2026-02-20T12:01:00.511 INFO:tasks.rados.rados.0.trial153.stdout:383: done (0 left) 2026-02-20T12:01:00.511 INFO:tasks.rados.rados.0.trial153.stdout:384: snap_remove snap 28 2026-02-20T12:01:00.969 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:01.046 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:01.039+0000 7f0a0aacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:01.048 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:01.043+0000 7f0a0aacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:01.074 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:01.090 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:01.513 INFO:tasks.rados.rados.0.trial153.stdout:384: done (0 left) 2026-02-20T12:01:01.513 INFO:tasks.rados.rados.0.trial153.stdout:385: snap_create 2026-02-20T12:01:02.516 INFO:tasks.rados.rados.0.trial153.stdout:385: done (0 left) 2026-02-20T12:01:02.516 INFO:tasks.rados.rados.0.trial153.stdout:386: read oid 32 snap -1 2026-02-20T12:01:02.516 INFO:tasks.rados.rados.0.trial153.stdout:386: expect (ObjNum 173 snap 39 seq_num 173) 2026-02-20T12:01:02.516 INFO:tasks.rados.rados.0.trial153.stdout:387: snap_remove snap 43 2026-02-20T12:01:03.091 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:03.168 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:03.162+0000 7fd1d9dd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:03.170 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:03.166+0000 7fd1d9dd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:03.196 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:03.212 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:03.524 INFO:tasks.rados.rados.0.trial153.stdout:387: done (1 left) 2026-02-20T12:01:03.524 INFO:tasks.rados.rados.0.trial153.stdout:388: snap_create 2026-02-20T12:01:03.524 INFO:tasks.rados.rados.0.trial153.stdout:386: expect (ObjNum 173 snap 39 seq_num 173) 2026-02-20T12:01:03.524 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:03.573 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 709710 2026-02-20T12:01:03.608 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:03.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~110s 2026-02-20T12:01:03.987 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:04.060 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:04.063 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:04.525 INFO:tasks.rados.rados.0.trial153.stdout:386: done (1 left) 2026-02-20T12:01:04.525 INFO:tasks.rados.rados.0.trial153.stdout:388: done (0 left) 2026-02-20T12:01:04.525 INFO:tasks.rados.rados.0.trial153.stdout:389: read oid 5 snap -1 2026-02-20T12:01:04.525 INFO:tasks.rados.rados.0.trial153.stdout:389: expect deleted 2026-02-20T12:01:04.526 INFO:tasks.rados.rados.0.trial153.stdout:390: write (excl) oid 10 current snap is 50 2026-02-20T12:01:04.526 INFO:tasks.rados.rados.0.trial153.stdout:390: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2026-02-20T12:01:04.529 INFO:tasks.rados.rados.0.trial153.stdout:390: writing trial15342161-10 from 561400 to 1036378 tid 1 2026-02-20T12:01:04.534 INFO:tasks.rados.rados.0.trial153.stdout:390: writing trial15342161-10 from 1820281 to 2340004 tid 2 2026-02-20T12:01:04.538 INFO:tasks.rados.rados.0.trial153.stdout:390: writing trial15342161-10 from 2859422 to 3317001 tid 3 2026-02-20T12:01:04.538 INFO:tasks.rados.rados.0.trial153.stdout:391: write oid 27 current snap is 50 2026-02-20T12:01:04.538 INFO:tasks.rados.rados.0.trial153.stdout:391: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2026-02-20T12:01:04.543 INFO:tasks.rados.rados.0.trial153.stdout:391: writing trial15342161-27 from 578207 to 1128434 tid 1 2026-02-20T12:01:04.548 INFO:tasks.rados.rados.0.trial153.stdout:391: writing trial15342161-27 from 1762410 to 2542144 tid 2 2026-02-20T12:01:04.551 INFO:tasks.rados.rados.0.trial153.stdout:391: writing trial15342161-27 from 3286845 to 3378207 tid 3 2026-02-20T12:01:04.552 INFO:tasks.rados.rados.0.trial153.stdout:392: read oid 35 snap -1 2026-02-20T12:01:04.552 INFO:tasks.rados.rados.0.trial153.stdout:392: expect deleted 2026-02-20T12:01:04.552 INFO:tasks.rados.rados.0.trial153.stdout:393: snap_create 2026-02-20T12:01:04.555 INFO:tasks.rados.rados.0.trial153.stdout:390: finishing write tid 1 to trial15342161-10 2026-02-20T12:01:04.571 INFO:tasks.rados.rados.0.trial153.stdout:391: finishing write tid 1 to trial15342161-27 2026-02-20T12:01:04.572 INFO:tasks.rados.rados.0.trial153.stdout:390: finishing write tid 2 to trial15342161-10 2026-02-20T12:01:04.577 INFO:tasks.rados.rados.0.trial153.stdout:391: finishing write tid 2 to trial15342161-27 2026-02-20T12:01:04.582 INFO:tasks.rados.rados.0.trial153.stdout:390: finishing write tid 3 to trial15342161-10 2026-02-20T12:01:04.584 INFO:tasks.rados.rados.0.trial153.stdout:391: finishing write tid 3 to trial15342161-27 2026-02-20T12:01:04.587 INFO:tasks.rados.rados.0.trial153.stdout:390: finishing write tid 4 to trial15342161-10 2026-02-20T12:01:04.589 INFO:tasks.rados.rados.0.trial153.stdout:391: finishing write tid 4 to trial15342161-27 2026-02-20T12:01:04.589 INFO:tasks.rados.rados.0.trial153.stdout:390: finishing write tid 5 to trial15342161-10 2026-02-20T12:01:04.590 INFO:tasks.rados.rados.0.trial153.stdout:390: oid 10 updating version 0 to 57 2026-02-20T12:01:04.590 INFO:tasks.rados.rados.0.trial153.stdout:390: oid 10 version 57 is already newer than 57 2026-02-20T12:01:04.590 INFO:tasks.rados.rados.0.trial153.stdout:390: oid 10 version 57 is already newer than 54 2026-02-20T12:01:04.590 INFO:tasks.rados.rados.0.trial153.stdout:390: oid 10 version 57 is already newer than 57 2026-02-20T12:01:04.590 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 10 v 57 (ObjNum 200 snap 50 seq_num 200) dirty exists 2026-02-20T12:01:04.590 INFO:tasks.rados.rados.0.trial153.stdout:390: left oid 10 (ObjNum 200 snap 50 seq_num 200) 2026-02-20T12:01:04.591 INFO:tasks.rados.rados.0.trial153.stdout:391: finishing write tid 5 to trial15342161-27 2026-02-20T12:01:04.591 INFO:tasks.rados.rados.0.trial153.stdout:391: oid 27 updating version 0 to 46 2026-02-20T12:01:04.591 INFO:tasks.rados.rados.0.trial153.stdout:391: oid 27 version 46 is already newer than 46 2026-02-20T12:01:04.591 INFO:tasks.rados.rados.0.trial153.stdout:391: oid 27 version 46 is already newer than 46 2026-02-20T12:01:04.591 INFO:tasks.rados.rados.0.trial153.stdout:391: oid 27 version 46 is already newer than 46 2026-02-20T12:01:04.591 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 27 v 46 (ObjNum 201 snap 50 seq_num 201) dirty exists 2026-02-20T12:01:04.592 INFO:tasks.rados.rados.0.trial153.stdout:391: left oid 27 (ObjNum 201 snap 50 seq_num 201) 2026-02-20T12:01:05.212 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:05.288 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:05.282+0000 7feb53c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:05.290 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:05.286+0000 7feb53c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:05.318 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:05.334 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:05.533 INFO:tasks.rados.rados.0.trial153.stdout:389: done (4 left) 2026-02-20T12:01:05.533 INFO:tasks.rados.rados.0.trial153.stdout:390: done (3 left) 2026-02-20T12:01:05.533 INFO:tasks.rados.rados.0.trial153.stdout:391: done (2 left) 2026-02-20T12:01:05.533 INFO:tasks.rados.rados.0.trial153.stdout:392: done (1 left) 2026-02-20T12:01:05.533 INFO:tasks.rados.rados.0.trial153.stdout:393: done (0 left) 2026-02-20T12:01:05.533 INFO:tasks.rados.rados.0.trial153.stdout:394: write (excl) oid 36 current snap is 51 2026-02-20T12:01:05.533 INFO:tasks.rados.rados.0.trial153.stdout:394: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2026-02-20T12:01:05.537 INFO:tasks.rados.rados.0.trial153.stdout:394: writing trial15342161-36 from 595014 to 1220490 tid 1 2026-02-20T12:01:05.541 INFO:tasks.rados.rados.0.trial153.stdout:394: writing trial15342161-36 from 1820892 to 2344284 tid 2 2026-02-20T12:01:05.546 INFO:tasks.rados.rados.0.trial153.stdout:394: writing trial15342161-36 from 2797915 to 3394391 tid 3 2026-02-20T12:01:05.546 INFO:tasks.rados.rados.0.trial153.stdout:395: read oid 12 snap -1 2026-02-20T12:01:05.546 INFO:tasks.rados.rados.0.trial153.stdout:395: expect deleted 2026-02-20T12:01:05.546 INFO:tasks.rados.rados.0.trial153.stdout:396: write (excl) oid 28 current snap is 51 2026-02-20T12:01:05.546 INFO:tasks.rados.rados.0.trial153.stdout:396: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2026-02-20T12:01:05.551 INFO:tasks.rados.rados.0.trial153.stdout:396: writing trial15342161-28 from 611821 to 1312546 tid 1 2026-02-20T12:01:05.556 INFO:tasks.rados.rados.0.trial153.stdout:396: writing trial15342161-28 from 1879374 to 2662777 tid 2 2026-02-20T12:01:05.559 INFO:tasks.rados.rados.0.trial153.stdout:396: writing trial15342161-28 from 3341691 to 3411821 tid 3 2026-02-20T12:01:05.559 INFO:tasks.rados.rados.0.trial153.stdout:397: write (excl) oid 50 current snap is 51 2026-02-20T12:01:05.559 INFO:tasks.rados.rados.0.trial153.stdout:397: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2026-02-20T12:01:05.564 INFO:tasks.rados.rados.0.trial153.stdout:397: writing trial15342161-50 from 628628 to 1404602 tid 1 2026-02-20T12:01:05.568 INFO:tasks.rados.rados.0.trial153.stdout:397: writing trial15342161-50 from 1937856 to 2464917 tid 2 2026-02-20T12:01:05.571 INFO:tasks.rados.rados.0.trial153.stdout:397: writing trial15342161-50 from 3252761 to 3428628 tid 3 2026-02-20T12:01:05.572 INFO:tasks.rados.rados.0.trial153.stdout:398: snap_create 2026-02-20T12:01:05.576 INFO:tasks.rados.rados.0.trial153.stdout:394: finishing write tid 1 to trial15342161-36 2026-02-20T12:01:05.589 INFO:tasks.rados.rados.0.trial153.stdout:396: finishing write tid 1 to trial15342161-28 2026-02-20T12:01:05.596 INFO:tasks.rados.rados.0.trial153.stdout:394: finishing write tid 2 to trial15342161-36 2026-02-20T12:01:05.604 INFO:tasks.rados.rados.0.trial153.stdout:394: finishing write tid 3 to trial15342161-36 2026-02-20T12:01:05.609 INFO:tasks.rados.rados.0.trial153.stdout:394: finishing write tid 4 to trial15342161-36 2026-02-20T12:01:05.611 INFO:tasks.rados.rados.0.trial153.stdout:394: finishing write tid 5 to trial15342161-36 2026-02-20T12:01:05.611 INFO:tasks.rados.rados.0.trial153.stdout:394: oid 36 updating version 0 to 46 2026-02-20T12:01:05.611 INFO:tasks.rados.rados.0.trial153.stdout:394: oid 36 version 46 is already newer than 46 2026-02-20T12:01:05.611 INFO:tasks.rados.rados.0.trial153.stdout:394: oid 36 version 46 is already newer than 46 2026-02-20T12:01:05.611 INFO:tasks.rados.rados.0.trial153.stdout:394: oid 36 version 46 is already newer than 43 2026-02-20T12:01:05.612 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 36 v 46 (ObjNum 202 snap 51 seq_num 202) dirty exists 2026-02-20T12:01:05.612 INFO:tasks.rados.rados.0.trial153.stdout:394: left oid 36 (ObjNum 202 snap 51 seq_num 202) 2026-02-20T12:01:05.612 INFO:tasks.rados.rados.0.trial153.stdout:396: finishing write tid 2 to trial15342161-28 2026-02-20T12:01:05.625 INFO:tasks.rados.rados.0.trial153.stdout:396: finishing write tid 3 to trial15342161-28 2026-02-20T12:01:05.632 INFO:tasks.rados.rados.0.trial153.stdout:396: finishing write tid 4 to trial15342161-28 2026-02-20T12:01:05.634 INFO:tasks.rados.rados.0.trial153.stdout:396: finishing write tid 5 to trial15342161-28 2026-02-20T12:01:05.634 INFO:tasks.rados.rados.0.trial153.stdout:396: oid 28 updating version 0 to 83 2026-02-20T12:01:05.634 INFO:tasks.rados.rados.0.trial153.stdout:396: oid 28 version 83 is already newer than 80 2026-02-20T12:01:05.634 INFO:tasks.rados.rados.0.trial153.stdout:396: oid 28 version 83 is already newer than 83 2026-02-20T12:01:05.634 INFO:tasks.rados.rados.0.trial153.stdout:396: oid 28 version 83 is already newer than 83 2026-02-20T12:01:05.634 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 28 v 83 (ObjNum 203 snap 51 seq_num 203) dirty exists 2026-02-20T12:01:05.634 INFO:tasks.rados.rados.0.trial153.stdout:396: left oid 28 (ObjNum 203 snap 51 seq_num 203) 2026-02-20T12:01:05.651 INFO:tasks.rados.rados.0.trial153.stdout:397: finishing write tid 1 to trial15342161-50 2026-02-20T12:01:05.665 INFO:tasks.rados.rados.0.trial153.stdout:397: finishing write tid 2 to trial15342161-50 2026-02-20T12:01:05.671 INFO:tasks.rados.rados.0.trial153.stdout:397: finishing write tid 3 to trial15342161-50 2026-02-20T12:01:05.675 INFO:tasks.rados.rados.0.trial153.stdout:397: finishing write tid 4 to trial15342161-50 2026-02-20T12:01:05.677 INFO:tasks.rados.rados.0.trial153.stdout:397: finishing write tid 5 to trial15342161-50 2026-02-20T12:01:05.677 INFO:tasks.rados.rados.0.trial153.stdout:397: oid 50 updating version 0 to 80 2026-02-20T12:01:05.677 INFO:tasks.rados.rados.0.trial153.stdout:397: oid 50 updating version 80 to 83 2026-02-20T12:01:05.677 INFO:tasks.rados.rados.0.trial153.stdout:397: oid 50 version 83 is already newer than 83 2026-02-20T12:01:05.677 INFO:tasks.rados.rados.0.trial153.stdout:397: oid 50 version 83 is already newer than 83 2026-02-20T12:01:05.677 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 50 v 83 (ObjNum 204 snap 51 seq_num 204) dirty exists 2026-02-20T12:01:05.677 INFO:tasks.rados.rados.0.trial153.stdout:397: left oid 50 (ObjNum 204 snap 51 seq_num 204) 2026-02-20T12:01:06.503 INFO:tasks.rados.rados.0.trial153.stdout:394: done (4 left) 2026-02-20T12:01:06.504 INFO:tasks.rados.rados.0.trial153.stdout:395: done (3 left) 2026-02-20T12:01:06.504 INFO:tasks.rados.rados.0.trial153.stdout:396: done (2 left) 2026-02-20T12:01:06.504 INFO:tasks.rados.rados.0.trial153.stdout:397: done (1 left) 2026-02-20T12:01:06.504 INFO:tasks.rados.rados.0.trial153.stdout:398: done (0 left) 2026-02-20T12:01:06.504 INFO:tasks.rados.rados.0.trial153.stdout:399: write oid 18 current snap is 52 2026-02-20T12:01:06.504 INFO:tasks.rados.rados.0.trial153.stdout:399: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2026-02-20T12:01:06.507 INFO:tasks.rados.rados.0.trial153.stdout:399: writing trial15342161-18 from 645435 to 1096658 tid 1 2026-02-20T12:01:06.514 INFO:tasks.rados.rados.0.trial153.stdout:399: writing trial15342161-18 from 1879985 to 2667057 tid 2 2026-02-20T12:01:06.517 INFO:tasks.rados.rados.0.trial153.stdout:399: writing trial15342161-18 from 3280184 to 3445435 tid 3 2026-02-20T12:01:06.517 INFO:tasks.rados.rados.0.trial153.stdout:400: read oid 47 snap -1 2026-02-20T12:01:06.517 INFO:tasks.rados.rados.0.trial153.stdout:400: expect (ObjNum 195 snap 47 seq_num 195) 2026-02-20T12:01:06.517 INFO:tasks.rados.rados.0.trial153.stdout:401: write (excl) oid 14 current snap is 52 2026-02-20T12:01:06.517 INFO:tasks.rados.rados.0.trial153.stdout:401: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2026-02-20T12:01:06.522 INFO:tasks.rados.rados.0.trial153.stdout:401: writing trial15342161-14 from 662242 to 1305067 tid 1 2026-02-20T12:01:06.526 INFO:tasks.rados.rados.0.trial153.stdout:401: writing trial15342161-14 from 2054820 to 2585550 tid 2 2026-02-20T12:01:06.530 INFO:tasks.rados.rados.0.trial153.stdout:401: writing trial15342161-14 from 3307607 to 3462242 tid 3 2026-02-20T12:01:06.530 INFO:tasks.rados.rados.0.trial153.stdout:402: delete oid 44 current snap is 52 2026-02-20T12:01:06.541 INFO:tasks.rados.rados.0.trial153.stdout:400: expect (ObjNum 195 snap 47 seq_num 195) 2026-02-20T12:01:06.576 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2171455 2026-02-20T12:01:06.589 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1175803 2026-02-20T12:01:06.608 INFO:tasks.rados.rados.0.trial153.stdout:399: finishing write tid 1 to trial15342161-18 2026-02-20T12:01:06.608 INFO:tasks.rados.rados.0.trial153.stdout:399: finishing write tid 2 to trial15342161-18 2026-02-20T12:01:06.608 INFO:tasks.rados.rados.0.trial153.stdout:401: finishing write tid 1 to trial15342161-14 2026-02-20T12:01:06.608 INFO:tasks.rados.rados.0.trial153.stdout:399: finishing write tid 3 to trial15342161-18 2026-02-20T12:01:06.608 INFO:tasks.rados.rados.0.trial153.stdout:399: finishing write tid 4 to trial15342161-18 2026-02-20T12:01:06.608 INFO:tasks.rados.rados.0.trial153.stdout:399: finishing write tid 5 to trial15342161-18 2026-02-20T12:01:06.608 INFO:tasks.rados.rados.0.trial153.stdout:399: oid 18 updating version 0 to 8 2026-02-20T12:01:06.608 INFO:tasks.rados.rados.0.trial153.stdout:399: oid 18 version 8 is already newer than 8 2026-02-20T12:01:06.608 INFO:tasks.rados.rados.0.trial153.stdout:399: oid 18 version 8 is already newer than 8 2026-02-20T12:01:06.609 INFO:tasks.rados.rados.0.trial153.stdout:399: oid 18 version 8 is already newer than 8 2026-02-20T12:01:06.609 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 18 v 8 (ObjNum 205 snap 52 seq_num 205) dirty exists 2026-02-20T12:01:06.609 INFO:tasks.rados.rados.0.trial153.stdout:399: left oid 18 (ObjNum 205 snap 52 seq_num 205) 2026-02-20T12:01:06.609 INFO:tasks.rados.rados.0.trial153.stdout:401: finishing write tid 2 to trial15342161-14 2026-02-20T12:01:06.609 INFO:tasks.rados.rados.0.trial153.stdout:401: finishing write tid 3 to trial15342161-14 2026-02-20T12:01:06.615 INFO:tasks.rados.rados.0.trial153.stdout:401: finishing write tid 4 to trial15342161-14 2026-02-20T12:01:06.617 INFO:tasks.rados.rados.0.trial153.stdout:401: finishing write tid 5 to trial15342161-14 2026-02-20T12:01:06.617 INFO:tasks.rados.rados.0.trial153.stdout:401: oid 14 updating version 0 to 88 2026-02-20T12:01:06.617 INFO:tasks.rados.rados.0.trial153.stdout:401: oid 14 version 88 is already newer than 88 2026-02-20T12:01:06.617 INFO:tasks.rados.rados.0.trial153.stdout:401: oid 14 version 88 is already newer than 88 2026-02-20T12:01:06.618 INFO:tasks.rados.rados.0.trial153.stdout:401: oid 14 version 88 is already newer than 85 2026-02-20T12:01:06.618 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 88 (ObjNum 206 snap 52 seq_num 206) dirty exists 2026-02-20T12:01:06.618 INFO:tasks.rados.rados.0.trial153.stdout:401: left oid 14 (ObjNum 206 snap 52 seq_num 206) 2026-02-20T12:01:06.629 INFO:tasks.rados.rados.0.trial153.stdout:399: done (3 left) 2026-02-20T12:01:06.629 INFO:tasks.rados.rados.0.trial153.stdout:400: done (2 left) 2026-02-20T12:01:06.630 INFO:tasks.rados.rados.0.trial153.stdout:401: done (1 left) 2026-02-20T12:01:06.630 INFO:tasks.rados.rados.0.trial153.stdout:402: done (0 left) 2026-02-20T12:01:06.630 INFO:tasks.rados.rados.0.trial153.stdout:403: read oid 40 snap 49 2026-02-20T12:01:06.630 INFO:tasks.rados.rados.0.trial153.stdout:403: expect (ObjNum 102 snap 13 seq_num 102) 2026-02-20T12:01:06.630 INFO:tasks.rados.rados.0.trial153.stdout:404: read oid 23 snap 51 2026-02-20T12:01:06.630 INFO:tasks.rados.rados.0.trial153.stdout:404: expect (ObjNum 169 snap 38 seq_num 169) 2026-02-20T12:01:06.630 INFO:tasks.rados.rados.0.trial153.stdout:405: snap_create 2026-02-20T12:01:06.655 INFO:tasks.rados.rados.0.trial153.stdout:403: expect (ObjNum 102 snap 13 seq_num 102) 2026-02-20T12:01:06.656 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:06.711 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 816406 2026-02-20T12:01:06.756 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 38481 2026-02-20T12:01:06.808 INFO:tasks.rados.rados.0.trial153.stdout:404: expect (ObjNum 169 snap 38 seq_num 169) 2026-02-20T12:01:06.813 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 916376 2026-02-20T12:01:06.877 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:07.335 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:07.415 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:07.409+0000 7f1324a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:07.417 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:07.413+0000 7f1324a46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:07.443 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:07.460 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:07.536 INFO:tasks.rados.rados.0.trial153.stdout:403: done (2 left) 2026-02-20T12:01:07.536 INFO:tasks.rados.rados.0.trial153.stdout:404: done (1 left) 2026-02-20T12:01:07.536 INFO:tasks.rados.rados.0.trial153.stdout:405: done (0 left) 2026-02-20T12:01:07.536 INFO:tasks.rados.rados.0.trial153.stdout:406: snap_create 2026-02-20T12:01:08.540 INFO:tasks.rados.rados.0.trial153.stdout:406: done (0 left) 2026-02-20T12:01:08.540 INFO:tasks.rados.rados.0.trial153.stdout:407: write (excl) oid 33 current snap is 54 2026-02-20T12:01:08.540 INFO:tasks.rados.rados.0.trial153.stdout:407: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2026-02-20T12:01:08.545 INFO:tasks.rados.rados.0.trial153.stdout:407: writing trial15342161-33 from 695856 to 1489179 tid 1 2026-02-20T12:01:08.550 INFO:tasks.rados.rados.0.trial153.stdout:407: writing trial15342161-33 from 2171784 to 2706183 tid 2 2026-02-20T12:01:08.553 INFO:tasks.rados.rados.0.trial153.stdout:407: writing trial15342161-33 from 3362453 to 3495856 tid 3 2026-02-20T12:01:08.553 INFO:tasks.rados.rados.0.trial153.stdout:408: write oid 9 current snap is 54 2026-02-20T12:01:08.553 INFO:tasks.rados.rados.0.trial153.stdout:408: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2026-02-20T12:01:08.557 INFO:tasks.rados.rados.0.trial153.stdout:408: writing trial15342161-9 from 712663 to 1181235 tid 1 2026-02-20T12:01:08.562 INFO:tasks.rados.rados.0.trial153.stdout:408: writing trial15342161-9 from 1713913 to 2391970 tid 2 2026-02-20T12:01:08.566 INFO:tasks.rados.rados.0.trial153.stdout:408: writing trial15342161-9 from 2873523 to 3465256 tid 3 2026-02-20T12:01:08.566 INFO:tasks.rados.rados.0.trial153.stdout:409: write (excl) oid 10 current snap is 54 2026-02-20T12:01:08.566 INFO:tasks.rados.rados.0.trial153.stdout:409: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2026-02-20T12:01:08.571 INFO:tasks.rados.rados.0.trial153.stdout:409: writing trial15342161-10 from 729470 to 1273291 tid 1 2026-02-20T12:01:08.574 INFO:tasks.rados.rados.0.trial153.stdout:409: writing trial15342161-10 from 1772395 to 2310463 tid 2 2026-02-20T12:01:08.578 INFO:tasks.rados.rados.0.trial153.stdout:409: writing trial15342161-10 from 2900946 to 3303951 tid 3 2026-02-20T12:01:08.579 INFO:tasks.rados.rados.0.trial153.stdout:410: write (excl) oid 18 current snap is 54 2026-02-20T12:01:08.579 INFO:tasks.rados.rados.0.trial153.stdout:410: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2026-02-20T12:01:08.583 INFO:tasks.rados.rados.0.trial153.stdout:410: writing trial15342161-18 from 746277 to 1365347 tid 1 2026-02-20T12:01:08.588 INFO:tasks.rados.rados.0.trial153.stdout:410: writing trial15342161-18 from 1830877 to 2512603 tid 2 2026-02-20T12:01:08.592 INFO:tasks.rados.rados.0.trial153.stdout:410: writing trial15342161-18 from 2928369 to 3546277 tid 3 2026-02-20T12:01:08.592 INFO:tasks.rados.rados.0.trial153.stdout:411: snap_remove snap 49 2026-02-20T12:01:09.065 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:09.137 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:09.140 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:09.140 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:09.393 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~116s 2026-02-20T12:01:09.461 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:09.538 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:09.532+0000 7f755615c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:09.543 INFO:tasks.rados.rados.0.trial153.stdout:411: done (4 left) 2026-02-20T12:01:09.543 INFO:tasks.rados.rados.0.trial153.stdout:412: read oid 30 snap 50 2026-02-20T12:01:09.543 INFO:tasks.rados.rados.0.trial153.stdout:412: expect deleted 2026-02-20T12:01:09.543 INFO:tasks.rados.rados.0.trial153.stdout:413: snap_create 2026-02-20T12:01:09.543 INFO:tasks.rados.rados.0.trial153.stdout:407: finishing write tid 1 to trial15342161-33 2026-02-20T12:01:09.543 INFO:tasks.rados.rados.0.trial153.stdout:407: finishing write tid 2 to trial15342161-33 2026-02-20T12:01:09.543 INFO:tasks.rados.rados.0.trial153.stdout:407: finishing write tid 3 to trial15342161-33 2026-02-20T12:01:09.543 INFO:tasks.rados.rados.0.trial153.stdout:407: finishing write tid 4 to trial15342161-33 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:407: finishing write tid 5 to trial15342161-33 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:407: oid 33 updating version 0 to 51 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:407: oid 33 version 51 is already newer than 51 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:407: oid 33 version 51 is already newer than 51 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:407: oid 33 version 51 is already newer than 48 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 33 v 51 (ObjNum 208 snap 54 seq_num 208) dirty exists 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:407: left oid 33 (ObjNum 208 snap 54 seq_num 208) 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:408: finishing write tid 1 to trial15342161-9 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:408: finishing write tid 2 to trial15342161-9 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:409: finishing write tid 1 to trial15342161-10 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:408: finishing write tid 3 to trial15342161-9 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:408: finishing write tid 4 to trial15342161-9 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:408: finishing write tid 5 to trial15342161-9 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:408: oid 9 updating version 0 to 151 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:408: oid 9 version 151 is already newer than 151 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:408: oid 9 version 151 is already newer than 151 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:408: oid 9 version 151 is already newer than 151 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 9 v 151 (ObjNum 209 snap 54 seq_num 209) dirty exists 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:408: left oid 9 (ObjNum 209 snap 54 seq_num 209) 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:409: finishing write tid 2 to trial15342161-10 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:409: finishing write tid 3 to trial15342161-10 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:409: finishing write tid 4 to trial15342161-10 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:409: finishing write tid 5 to trial15342161-10 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:409: oid 10 updating version 0 to 64 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:409: oid 10 version 64 is already newer than 64 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:409: oid 10 version 64 is already newer than 61 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:409: oid 10 version 64 is already newer than 64 2026-02-20T12:01:09.544 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 10 v 64 (ObjNum 210 snap 54 seq_num 210) dirty exists 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:409: left oid 10 (ObjNum 210 snap 54 seq_num 210) 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:410: finishing write tid 1 to trial15342161-18 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:410: finishing write tid 2 to trial15342161-18 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:410: finishing write tid 3 to trial15342161-18 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:410: finishing write tid 4 to trial15342161-18 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:410: finishing write tid 5 to trial15342161-18 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:410: oid 18 updating version 0 to 13 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:410: oid 18 version 13 is already newer than 10 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:410: oid 18 version 13 is already newer than 13 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:410: oid 18 version 13 is already newer than 13 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 18 v 13 (ObjNum 211 snap 54 seq_num 211) dirty exists 2026-02-20T12:01:09.545 INFO:tasks.rados.rados.0.trial153.stdout:410: left oid 18 (ObjNum 211 snap 54 seq_num 211) 2026-02-20T12:01:09.545 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:09.539+0000 7f755615c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:09.591 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:09.605 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:09.606 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:10.545 INFO:tasks.rados.rados.0.trial153.stdout:407: done (5 left) 2026-02-20T12:01:10.546 INFO:tasks.rados.rados.0.trial153.stdout:408: done (4 left) 2026-02-20T12:01:10.546 INFO:tasks.rados.rados.0.trial153.stdout:409: done (3 left) 2026-02-20T12:01:10.546 INFO:tasks.rados.rados.0.trial153.stdout:410: done (2 left) 2026-02-20T12:01:10.546 INFO:tasks.rados.rados.0.trial153.stdout:412: done (1 left) 2026-02-20T12:01:10.546 INFO:tasks.rados.rados.0.trial153.stdout:413: done (0 left) 2026-02-20T12:01:10.546 INFO:tasks.rados.rados.0.trial153.stdout:414: write (excl) oid 1 current snap is 55 2026-02-20T12:01:10.546 INFO:tasks.rados.rados.0.trial153.stdout:414: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2026-02-20T12:01:10.551 INFO:tasks.rados.rados.0.trial153.stdout:414: writing trial15342161-1 from 763084 to 1457403 tid 1 2026-02-20T12:01:10.555 INFO:tasks.rados.rados.0.trial153.stdout:414: writing trial15342161-1 from 1889359 to 2431096 tid 2 2026-02-20T12:01:10.560 INFO:tasks.rados.rados.0.trial153.stdout:414: writing trial15342161-1 from 2955792 to 3497694 tid 3 2026-02-20T12:01:10.560 INFO:tasks.rados.rados.0.trial153.stdout:415: read oid 10 snap 54 2026-02-20T12:01:10.560 INFO:tasks.rados.rados.0.trial153.stdout:415: expect (ObjNum 210 snap 54 seq_num 210) 2026-02-20T12:01:10.560 INFO:tasks.rados.rados.0.trial153.stdout:416: snap_create 2026-02-20T12:01:10.574 INFO:tasks.rados.rados.0.trial153.stdout:414: finishing write tid 1 to trial15342161-1 2026-02-20T12:01:10.587 INFO:tasks.rados.rados.0.trial153.stdout:414: finishing write tid 2 to trial15342161-1 2026-02-20T12:01:10.594 INFO:tasks.rados.rados.0.trial153.stdout:414: finishing write tid 3 to trial15342161-1 2026-02-20T12:01:10.599 INFO:tasks.rados.rados.0.trial153.stdout:414: finishing write tid 4 to trial15342161-1 2026-02-20T12:01:10.600 INFO:tasks.rados.rados.0.trial153.stdout:414: finishing write tid 5 to trial15342161-1 2026-02-20T12:01:10.601 INFO:tasks.rados.rados.0.trial153.stdout:414: oid 1 updating version 0 to 157 2026-02-20T12:01:10.601 INFO:tasks.rados.rados.0.trial153.stdout:414: oid 1 version 157 is already newer than 157 2026-02-20T12:01:10.601 INFO:tasks.rados.rados.0.trial153.stdout:414: oid 1 version 157 is already newer than 157 2026-02-20T12:01:10.601 INFO:tasks.rados.rados.0.trial153.stdout:414: oid 1 version 157 is already newer than 154 2026-02-20T12:01:10.601 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 1 v 157 (ObjNum 212 snap 55 seq_num 212) dirty exists 2026-02-20T12:01:10.601 INFO:tasks.rados.rados.0.trial153.stdout:414: left oid 1 (ObjNum 212 snap 55 seq_num 212) 2026-02-20T12:01:10.601 INFO:tasks.rados.rados.0.trial153.stdout:415: expect (ObjNum 210 snap 54 seq_num 210) 2026-02-20T12:01:11.552 INFO:tasks.rados.rados.0.trial153.stdout:414: done (2 left) 2026-02-20T12:01:11.553 INFO:tasks.rados.rados.0.trial153.stdout:415: done (1 left) 2026-02-20T12:01:11.553 INFO:tasks.rados.rados.0.trial153.stdout:416: done (0 left) 2026-02-20T12:01:11.553 INFO:tasks.rados.rados.0.trial153.stdout:417: read oid 6 snap 55 2026-02-20T12:01:11.553 INFO:tasks.rados.rados.0.trial153.stdout:417: expect deleted 2026-02-20T12:01:11.553 INFO:tasks.rados.rados.0.trial153.stdout:418: write (excl) oid 4 current snap is 56 2026-02-20T12:01:11.553 INFO:tasks.rados.rados.0.trial153.stdout:418: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2026-02-20T12:01:11.558 INFO:tasks.rados.rados.0.trial153.stdout:418: writing trial15342161-4 from 779891 to 1265812 tid 1 2026-02-20T12:01:11.564 INFO:tasks.rados.rados.0.trial153.stdout:418: writing trial15342161-4 from 1947841 to 2633236 tid 2 2026-02-20T12:01:11.568 INFO:tasks.rados.rados.0.trial153.stdout:418: writing trial15342161-4 from 3383215 to 3579891 tid 3 2026-02-20T12:01:11.568 INFO:tasks.rados.rados.0.trial153.stdout:419: write (excl) oid 44 current snap is 56 2026-02-20T12:01:11.568 INFO:tasks.rados.rados.0.trial153.stdout:419: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2026-02-20T12:01:11.573 INFO:tasks.rados.rados.0.trial153.stdout:419: writing trial15342161-44 from 796698 to 1357868 tid 1 2026-02-20T12:01:11.578 INFO:tasks.rados.rados.0.trial153.stdout:419: writing trial15342161-44 from 2006323 to 2551729 tid 2 2026-02-20T12:01:11.582 INFO:tasks.rados.rados.0.trial153.stdout:419: writing trial15342161-44 from 3126991 to 3596698 tid 3 2026-02-20T12:01:11.582 INFO:tasks.rados.rados.0.trial153.stdout:420: delete oid 26 current snap is 56 2026-02-20T12:01:11.588 INFO:tasks.rados.rados.0.trial153.stdout:419: finishing write tid 1 to trial15342161-44 2026-02-20T12:01:11.604 INFO:tasks.rados.rados.0.trial153.stdout:419: finishing write tid 2 to trial15342161-44 2026-02-20T12:01:11.606 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:11.607 INFO:tasks.rados.rados.0.trial153.stdout:418: finishing write tid 1 to trial15342161-4 2026-02-20T12:01:11.617 INFO:tasks.rados.rados.0.trial153.stdout:419: finishing write tid 3 to trial15342161-44 2026-02-20T12:01:11.621 INFO:tasks.rados.rados.0.trial153.stdout:419: finishing write tid 4 to trial15342161-44 2026-02-20T12:01:11.624 INFO:tasks.rados.rados.0.trial153.stdout:419: finishing write tid 5 to trial15342161-44 2026-02-20T12:01:11.624 INFO:tasks.rados.rados.0.trial153.stdout:419: oid 44 updating version 0 to 65 2026-02-20T12:01:11.624 INFO:tasks.rados.rados.0.trial153.stdout:419: oid 44 updating version 65 to 68 2026-02-20T12:01:11.624 INFO:tasks.rados.rados.0.trial153.stdout:419: oid 44 version 68 is already newer than 68 2026-02-20T12:01:11.624 INFO:tasks.rados.rados.0.trial153.stdout:419: oid 44 version 68 is already newer than 68 2026-02-20T12:01:11.625 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 44 v 68 (ObjNum 214 snap 56 seq_num 214) dirty exists 2026-02-20T12:01:11.625 INFO:tasks.rados.rados.0.trial153.stdout:419: left oid 44 (ObjNum 214 snap 56 seq_num 214) 2026-02-20T12:01:11.641 INFO:tasks.rados.rados.0.trial153.stdout:418: finishing write tid 2 to trial15342161-4 2026-02-20T12:01:11.651 INFO:tasks.rados.rados.0.trial153.stdout:418: finishing write tid 3 to trial15342161-4 2026-02-20T12:01:11.659 INFO:tasks.rados.rados.0.trial153.stdout:418: finishing write tid 4 to trial15342161-4 2026-02-20T12:01:11.660 INFO:tasks.rados.rados.0.trial153.stdout:418: finishing write tid 5 to trial15342161-4 2026-02-20T12:01:11.660 INFO:tasks.rados.rados.0.trial153.stdout:418: oid 4 updating version 0 to 73 2026-02-20T12:01:11.660 INFO:tasks.rados.rados.0.trial153.stdout:418: oid 4 version 73 is already newer than 73 2026-02-20T12:01:11.660 INFO:tasks.rados.rados.0.trial153.stdout:418: oid 4 version 73 is already newer than 73 2026-02-20T12:01:11.660 INFO:tasks.rados.rados.0.trial153.stdout:418: oid 4 version 73 is already newer than 70 2026-02-20T12:01:11.660 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 4 v 73 (ObjNum 213 snap 56 seq_num 213) dirty exists 2026-02-20T12:01:11.660 INFO:tasks.rados.rados.0.trial153.stdout:418: left oid 4 (ObjNum 213 snap 56 seq_num 213) 2026-02-20T12:01:11.681 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:11.676+0000 7fc6bf61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:11.684 INFO:tasks.rados.rados.0.trial153.stdout:417: done (3 left) 2026-02-20T12:01:11.684 INFO:tasks.rados.rados.0.trial153.stdout:418: done (2 left) 2026-02-20T12:01:11.684 INFO:tasks.rados.rados.0.trial153.stdout:419: done (1 left) 2026-02-20T12:01:11.684 INFO:tasks.rados.rados.0.trial153.stdout:420: done (0 left) 2026-02-20T12:01:11.684 INFO:tasks.rados.rados.0.trial153.stdout:421: read oid 31 snap -1 2026-02-20T12:01:11.684 INFO:tasks.rados.rados.0.trial153.stdout:421: expect (ObjNum 128 snap 19 seq_num 128) 2026-02-20T12:01:11.684 INFO:tasks.rados.rados.0.trial153.stdout:422: read oid 32 snap 51 2026-02-20T12:01:11.684 INFO:tasks.rados.rados.0.trial153.stdout:422: expect (ObjNum 173 snap 39 seq_num 173) 2026-02-20T12:01:11.684 INFO:tasks.rados.rados.0.trial153.stdout:423: delete oid 22 current snap is 56 2026-02-20T12:01:11.685 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:11.680+0000 7fc6bf61a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:11.699 INFO:tasks.rados.rados.0.trial153.stdout:421: expect (ObjNum 128 snap 19 seq_num 128) 2026-02-20T12:01:11.699 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:11.713 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:11.725 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 171345 2026-02-20T12:01:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:11.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:11.764 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2075499 2026-02-20T12:01:11.764 INFO:tasks.rados.rados.0.trial153.stdout:422: expect (ObjNum 173 snap 39 seq_num 173) 2026-02-20T12:01:11.877 INFO:tasks.rados.rados.0.trial153.stdout:421: done (2 left) 2026-02-20T12:01:11.877 INFO:tasks.rados.rados.0.trial153.stdout:422: done (1 left) 2026-02-20T12:01:11.877 INFO:tasks.rados.rados.0.trial153.stdout:423: done (0 left) 2026-02-20T12:01:11.877 INFO:tasks.rados.rados.0.trial153.stdout:424: read oid 21 snap -1 2026-02-20T12:01:11.877 INFO:tasks.rados.rados.0.trial153.stdout:424: expect deleted 2026-02-20T12:01:11.878 INFO:tasks.rados.rados.0.trial153.stdout:425: write oid 2 current snap is 56 2026-02-20T12:01:11.878 INFO:tasks.rados.rados.0.trial153.stdout:425: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2026-02-20T12:01:11.881 INFO:tasks.rados.rados.0.trial153.stdout:425: writing trial15342161-2 from 447119 to 1234036 tid 1 2026-02-20T12:01:11.885 INFO:tasks.rados.rados.0.trial153.stdout:425: writing trial15342161-2 from 1665416 to 2358149 tid 2 2026-02-20T12:01:11.890 INFO:tasks.rados.rados.0.trial153.stdout:425: writing trial15342161-2 from 2976554 to 3491169 tid 3 2026-02-20T12:01:11.890 INFO:tasks.rados.rados.0.trial153.stdout:426: write (excl) oid 10 current snap is 56 2026-02-20T12:01:11.890 INFO:tasks.rados.rados.0.trial153.stdout:426: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2026-02-20T12:01:11.893 INFO:tasks.rados.rados.0.trial153.stdout:426: writing trial15342161-10 from 463926 to 926092 tid 1 2026-02-20T12:01:11.898 INFO:tasks.rados.rados.0.trial153.stdout:426: writing trial15342161-10 from 1723898 to 2160289 tid 2 2026-02-20T12:01:11.902 INFO:tasks.rados.rados.0.trial153.stdout:426: writing trial15342161-10 from 2603977 to 3329864 tid 3 2026-02-20T12:01:11.902 INFO:tasks.rados.rados.0.trial153.stdout:427: snap_remove snap 54 2026-02-20T12:01:12.556 INFO:tasks.rados.rados.0.trial153.stdout:427: done (3 left) 2026-02-20T12:01:12.556 INFO:tasks.rados.rados.0.trial153.stdout:428: delete oid 36 current snap is 56 2026-02-20T12:01:12.556 INFO:tasks.rados.rados.0.trial153.stdout:425: finishing write tid 1 to trial15342161-2 2026-02-20T12:01:12.556 INFO:tasks.rados.rados.0.trial153.stdout:425: finishing write tid 2 to trial15342161-2 2026-02-20T12:01:12.556 INFO:tasks.rados.rados.0.trial153.stdout:425: finishing write tid 3 to trial15342161-2 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:425: finishing write tid 4 to trial15342161-2 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:426: finishing write tid 1 to trial15342161-10 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:425: finishing write tid 5 to trial15342161-2 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:425: oid 2 updating version 0 to 161 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:425: oid 2 version 161 is already newer than 161 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:425: oid 2 version 161 is already newer than 161 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:425: oid 2 version 161 is already newer than 161 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 2 v 161 (ObjNum 217 snap 56 seq_num 217) dirty exists 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:425: left oid 2 (ObjNum 217 snap 56 seq_num 217) 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:426: finishing write tid 2 to trial15342161-10 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:426: finishing write tid 3 to trial15342161-10 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:426: finishing write tid 4 to trial15342161-10 2026-02-20T12:01:12.557 INFO:tasks.rados.rados.0.trial153.stdout:426: finishing write tid 5 to trial15342161-10 2026-02-20T12:01:12.558 INFO:tasks.rados.rados.0.trial153.stdout:426: oid 10 updating version 0 to 73 2026-02-20T12:01:12.558 INFO:tasks.rados.rados.0.trial153.stdout:426: oid 10 version 73 is already newer than 73 2026-02-20T12:01:12.558 INFO:tasks.rados.rados.0.trial153.stdout:426: oid 10 version 73 is already newer than 73 2026-02-20T12:01:12.558 INFO:tasks.rados.rados.0.trial153.stdout:426: oid 10 version 73 is already newer than 70 2026-02-20T12:01:12.558 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 10 v 73 (ObjNum 218 snap 56 seq_num 218) dirty exists 2026-02-20T12:01:12.558 INFO:tasks.rados.rados.0.trial153.stdout:426: left oid 10 (ObjNum 218 snap 56 seq_num 218) 2026-02-20T12:01:12.610 INFO:tasks.rados.rados.0.trial153.stdout:424: done (3 left) 2026-02-20T12:01:12.610 INFO:tasks.rados.rados.0.trial153.stdout:425: done (2 left) 2026-02-20T12:01:12.611 INFO:tasks.rados.rados.0.trial153.stdout:426: done (1 left) 2026-02-20T12:01:12.611 INFO:tasks.rados.rados.0.trial153.stdout:428: done (0 left) 2026-02-20T12:01:12.611 INFO:tasks.rados.rados.0.trial153.stdout:429: read oid 39 snap 50 2026-02-20T12:01:12.611 INFO:tasks.rados.rados.0.trial153.stdout:429: expect deleted 2026-02-20T12:01:12.611 INFO:tasks.rados.rados.0.trial153.stdout:430: read oid 24 snap -1 2026-02-20T12:01:12.611 INFO:tasks.rados.rados.0.trial153.stdout:430: expect (ObjNum 181 snap 41 seq_num 181) 2026-02-20T12:01:12.611 INFO:tasks.rados.rados.0.trial153.stdout:431: write oid 35 current snap is 56 2026-02-20T12:01:12.611 INFO:tasks.rados.rados.0.trial153.stdout:431: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2026-02-20T12:01:12.615 INFO:tasks.rados.rados.0.trial153.stdout:431: writing trial15342161-35 from 497540 to 1110204 tid 1 2026-02-20T12:01:12.619 INFO:tasks.rados.rados.0.trial153.stdout:431: writing trial15342161-35 from 1840862 to 2280922 tid 2 2026-02-20T12:01:12.624 INFO:tasks.rados.rados.0.trial153.stdout:431: writing trial15342161-35 from 3058823 to 3523607 tid 3 2026-02-20T12:01:12.624 INFO:tasks.rados.rados.0.trial153.stdout:432: write oid 45 current snap is 56 2026-02-20T12:01:12.625 INFO:tasks.rados.rados.0.trial153.stdout:432: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2026-02-20T12:01:12.628 INFO:tasks.rados.rados.0.trial153.stdout:432: writing trial15342161-45 from 514347 to 918613 tid 1 2026-02-20T12:01:12.633 INFO:tasks.rados.rados.0.trial153.stdout:432: writing trial15342161-45 from 1499344 to 2199415 tid 2 2026-02-20T12:01:12.637 INFO:tasks.rados.rados.0.trial153.stdout:432: writing trial15342161-45 from 2686246 to 3362302 tid 3 2026-02-20T12:01:12.637 INFO:tasks.rados.rados.0.trial153.stdout:433: snap_remove snap 55 2026-02-20T12:01:13.559 INFO:tasks.rados.rados.0.trial153.stdout:433: done (4 left) 2026-02-20T12:01:13.559 INFO:tasks.rados.rados.0.trial153.stdout:434: read oid 14 snap 52 2026-02-20T12:01:13.559 INFO:tasks.rados.rados.0.trial153.stdout:434: expect (ObjNum 206 snap 52 seq_num 206) 2026-02-20T12:01:13.560 INFO:tasks.rados.rados.0.trial153.stdout:435: write oid 4 current snap is 56 2026-02-20T12:01:13.560 INFO:tasks.rados.rados.0.trial153.stdout:435: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2026-02-20T12:01:13.563 INFO:tasks.rados.rados.0.trial153.stdout:435: writing trial15342161-4 from 531154 to 1010669 tid 1 2026-02-20T12:01:13.567 INFO:tasks.rados.rados.0.trial153.stdout:435: writing trial15342161-4 from 1557826 to 2001555 tid 2 2026-02-20T12:01:13.571 INFO:tasks.rados.rados.0.trial153.stdout:435: writing trial15342161-4 from 2713669 to 3200997 tid 3 2026-02-20T12:01:13.574 INFO:tasks.rados.rados.0.trial153.stdout:435: writing trial15342161-4 from 3679295 to 3731154 tid 4 2026-02-20T12:01:13.574 INFO:tasks.rados.rados.0.trial153.stdout:436: snap_remove snap 51 2026-02-20T12:01:13.729 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:13.808 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:13.802+0000 7f8a5d9e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:13.810 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:13.806+0000 7f8a5d9e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:13.838 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:13.854 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:13.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:14.141 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:14.213 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:14.216 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:14.216 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:14.564 INFO:tasks.rados.rados.0.trial153.stdout:436: done (6 left) 2026-02-20T12:01:14.564 INFO:tasks.rados.rados.0.trial153.stdout:437: snap_create 2026-02-20T12:01:14.564 INFO:tasks.rados.rados.0.trial153.stdout:430: expect (ObjNum 181 snap 41 seq_num 181) 2026-02-20T12:01:14.566 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1996632 2026-02-20T12:01:14.581 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:14.611 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 878899 2026-02-20T12:01:14.629 INFO:tasks.rados.rados.0.trial153.stdout:431: finishing write tid 1 to trial15342161-35 2026-02-20T12:01:14.630 INFO:tasks.rados.rados.0.trial153.stdout:431: finishing write tid 2 to trial15342161-35 2026-02-20T12:01:14.630 INFO:tasks.rados.rados.0.trial153.stdout:431: finishing write tid 3 to trial15342161-35 2026-02-20T12:01:14.630 INFO:tasks.rados.rados.0.trial153.stdout:431: finishing write tid 4 to trial15342161-35 2026-02-20T12:01:14.630 INFO:tasks.rados.rados.0.trial153.stdout:431: finishing write tid 5 to trial15342161-35 2026-02-20T12:01:14.630 INFO:tasks.rados.rados.0.trial153.stdout:431: oid 35 updating version 0 to 89 2026-02-20T12:01:14.630 INFO:tasks.rados.rados.0.trial153.stdout:431: oid 35 version 89 is already newer than 89 2026-02-20T12:01:14.630 INFO:tasks.rados.rados.0.trial153.stdout:431: oid 35 version 89 is already newer than 89 2026-02-20T12:01:14.630 INFO:tasks.rados.rados.0.trial153.stdout:431: oid 35 version 89 is already newer than 89 2026-02-20T12:01:14.630 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 35 v 89 (ObjNum 220 snap 56 seq_num 220) dirty exists 2026-02-20T12:01:14.630 INFO:tasks.rados.rados.0.trial153.stdout:431: left oid 35 (ObjNum 220 snap 56 seq_num 220) 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:432: finishing write tid 1 to trial15342161-45 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:432: finishing write tid 2 to trial15342161-45 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:432: finishing write tid 3 to trial15342161-45 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:432: finishing write tid 4 to trial15342161-45 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:432: finishing write tid 5 to trial15342161-45 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:432: oid 45 updating version 0 to 46 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:432: oid 45 version 46 is already newer than 46 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:432: oid 45 version 46 is already newer than 46 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:432: oid 45 version 46 is already newer than 46 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 46 (ObjNum 221 snap 56 seq_num 221) dirty exists 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:432: left oid 45 (ObjNum 221 snap 56 seq_num 221) 2026-02-20T12:01:14.631 INFO:tasks.rados.rados.0.trial153.stdout:434: expect (ObjNum 206 snap 52 seq_num 206) 2026-02-20T12:01:14.672 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 3079687 2026-02-20T12:01:14.710 INFO:tasks.rados.rados.0.trial153.stdout:435: finishing write tid 1 to trial15342161-4 2026-02-20T12:01:14.710 INFO:tasks.rados.rados.0.trial153.stdout:435: finishing write tid 2 to trial15342161-4 2026-02-20T12:01:14.710 INFO:tasks.rados.rados.0.trial153.stdout:435: finishing write tid 3 to trial15342161-4 2026-02-20T12:01:14.710 INFO:tasks.rados.rados.0.trial153.stdout:435: finishing write tid 4 to trial15342161-4 2026-02-20T12:01:14.710 INFO:tasks.rados.rados.0.trial153.stdout:435: finishing write tid 5 to trial15342161-4 2026-02-20T12:01:14.710 INFO:tasks.rados.rados.0.trial153.stdout:435: finishing write tid 6 to trial15342161-4 2026-02-20T12:01:14.710 INFO:tasks.rados.rados.0.trial153.stdout:435: oid 4 updating version 0 to 78 2026-02-20T12:01:14.711 INFO:tasks.rados.rados.0.trial153.stdout:435: oid 4 version 78 is already newer than 78 2026-02-20T12:01:14.711 INFO:tasks.rados.rados.0.trial153.stdout:435: oid 4 version 78 is already newer than 78 2026-02-20T12:01:14.711 INFO:tasks.rados.rados.0.trial153.stdout:435: oid 4 version 78 is already newer than 78 2026-02-20T12:01:14.711 INFO:tasks.rados.rados.0.trial153.stdout:435: oid 4 version 78 is already newer than 78 2026-02-20T12:01:14.711 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 4 v 78 (ObjNum 222 snap 56 seq_num 222) dirty exists 2026-02-20T12:01:14.711 INFO:tasks.rados.rados.0.trial153.stdout:435: left oid 4 (ObjNum 222 snap 56 seq_num 222) 2026-02-20T12:01:14.896 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~121s 2026-02-20T12:01:15.567 INFO:tasks.rados.rados.0.trial153.stdout:429: done (6 left) 2026-02-20T12:01:15.567 INFO:tasks.rados.rados.0.trial153.stdout:430: done (5 left) 2026-02-20T12:01:15.567 INFO:tasks.rados.rados.0.trial153.stdout:431: done (4 left) 2026-02-20T12:01:15.567 INFO:tasks.rados.rados.0.trial153.stdout:432: done (3 left) 2026-02-20T12:01:15.567 INFO:tasks.rados.rados.0.trial153.stdout:434: done (2 left) 2026-02-20T12:01:15.567 INFO:tasks.rados.rados.0.trial153.stdout:435: done (1 left) 2026-02-20T12:01:15.567 INFO:tasks.rados.rados.0.trial153.stdout:437: done (0 left) 2026-02-20T12:01:15.567 INFO:tasks.rados.rados.0.trial153.stdout:438: snap_create 2026-02-20T12:01:15.856 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:15.935 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:15.929+0000 7febc0bd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:15.937 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:15.933+0000 7febc0bd3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:15.964 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:15.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:15.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:16.506 INFO:tasks.rados.rados.0.trial153.stdout:438: done (0 left) 2026-02-20T12:01:16.506 INFO:tasks.rados.rados.0.trial153.stdout:439: snap_remove snap 57 2026-02-20T12:01:17.507 INFO:tasks.rados.rados.0.trial153.stdout:439: done (0 left) 2026-02-20T12:01:17.507 INFO:tasks.rados.rados.0.trial153.stdout:440: write oid 46 current snap is 58 2026-02-20T12:01:17.507 INFO:tasks.rados.rados.0.trial153.stdout:440: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2026-02-20T12:01:17.511 INFO:tasks.rados.rados.0.trial153.stdout:440: writing trial15342161-46 from 547961 to 1102725 tid 1 2026-02-20T12:01:17.515 INFO:tasks.rados.rados.0.trial153.stdout:440: writing trial15342161-46 from 1616308 to 2320048 tid 2 2026-02-20T12:01:17.520 INFO:tasks.rados.rados.0.trial153.stdout:440: writing trial15342161-46 from 2741092 to 3439692 tid 3 2026-02-20T12:01:17.521 INFO:tasks.rados.rados.0.trial153.stdout:441: snap_remove snap 53 2026-02-20T12:01:17.981 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:18.061 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:18.055+0000 7ff8a5b74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:18.064 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:18.059+0000 7ff8a5b74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:18.090 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:18.105 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:18.511 INFO:tasks.rados.rados.0.trial153.stdout:441: done (1 left) 2026-02-20T12:01:18.511 INFO:tasks.rados.rados.0.trial153.stdout:442: delete oid 37 current snap is 58 2026-02-20T12:01:18.511 INFO:tasks.rados.rados.0.trial153.stdout:440: finishing write tid 1 to trial15342161-46 2026-02-20T12:01:18.511 INFO:tasks.rados.rados.0.trial153.stdout:440: finishing write tid 2 to trial15342161-46 2026-02-20T12:01:18.511 INFO:tasks.rados.rados.0.trial153.stdout:440: finishing write tid 3 to trial15342161-46 2026-02-20T12:01:18.511 INFO:tasks.rados.rados.0.trial153.stdout:440: finishing write tid 4 to trial15342161-46 2026-02-20T12:01:18.511 INFO:tasks.rados.rados.0.trial153.stdout:440: finishing write tid 5 to trial15342161-46 2026-02-20T12:01:18.511 INFO:tasks.rados.rados.0.trial153.stdout:440: oid 46 updating version 0 to 18 2026-02-20T12:01:18.511 INFO:tasks.rados.rados.0.trial153.stdout:440: oid 46 version 18 is already newer than 18 2026-02-20T12:01:18.512 INFO:tasks.rados.rados.0.trial153.stdout:440: oid 46 version 18 is already newer than 18 2026-02-20T12:01:18.512 INFO:tasks.rados.rados.0.trial153.stdout:440: oid 46 version 18 is already newer than 18 2026-02-20T12:01:18.512 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 46 v 18 (ObjNum 223 snap 58 seq_num 223) dirty exists 2026-02-20T12:01:18.512 INFO:tasks.rados.rados.0.trial153.stdout:440: left oid 46 (ObjNum 223 snap 58 seq_num 223) 2026-02-20T12:01:18.564 INFO:tasks.rados.rados.0.trial153.stdout:440: done (1 left) 2026-02-20T12:01:18.565 INFO:tasks.rados.rados.0.trial153.stdout:442: done (0 left) 2026-02-20T12:01:18.565 INFO:tasks.rados.rados.0.trial153.stdout:443: delete oid 48 current snap is 58 2026-02-20T12:01:18.574 INFO:tasks.rados.rados.0.trial153.stdout:443: done (0 left) 2026-02-20T12:01:18.574 INFO:tasks.rados.rados.0.trial153.stdout:444: read oid 9 snap -1 2026-02-20T12:01:18.574 INFO:tasks.rados.rados.0.trial153.stdout:444: expect (ObjNum 209 snap 54 seq_num 209) 2026-02-20T12:01:18.574 INFO:tasks.rados.rados.0.trial153.stdout:445: read oid 33 snap 52 2026-02-20T12:01:18.574 INFO:tasks.rados.rados.0.trial153.stdout:445: expect (ObjNum 165 snap 35 seq_num 165) 2026-02-20T12:01:18.574 INFO:tasks.rados.rados.0.trial153.stdout:446: write oid 44 current snap is 58 2026-02-20T12:01:18.574 INFO:tasks.rados.rados.0.trial153.stdout:446: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2026-02-20T12:01:18.579 INFO:tasks.rados.rados.0.trial153.stdout:446: writing trial15342161-44 from 598382 to 1378893 tid 1 2026-02-20T12:01:18.583 INFO:tasks.rados.rados.0.trial153.stdout:446: writing trial15342161-44 from 2075401 to 2526468 tid 2 2026-02-20T12:01:18.588 INFO:tasks.rados.rados.0.trial153.stdout:446: writing trial15342161-44 from 3107008 to 3755777 tid 3 2026-02-20T12:01:18.588 INFO:tasks.rados.rados.0.trial153.stdout:447: read oid 47 snap 56 2026-02-20T12:01:18.588 INFO:tasks.rados.rados.0.trial153.stdout:447: expect (ObjNum 195 snap 47 seq_num 195) 2026-02-20T12:01:18.588 INFO:tasks.rados.rados.0.trial153.stdout:448: snap_create 2026-02-20T12:01:18.595 INFO:tasks.rados.rados.0.trial153.stdout:445: expect (ObjNum 165 snap 35 seq_num 165) 2026-02-20T12:01:18.610 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:18.645 INFO:tasks.rados.rados.0.trial153.stdout:444: expect (ObjNum 209 snap 54 seq_num 209) 2026-02-20T12:01:18.726 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1606136 2026-02-20T12:01:18.761 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:18.824 INFO:tasks.rados.rados.0.trial153.stdout:447: expect (ObjNum 195 snap 47 seq_num 195) 2026-02-20T12:01:18.825 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:18.879 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:446: finishing write tid 1 to trial15342161-44 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:446: finishing write tid 2 to trial15342161-44 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:446: finishing write tid 3 to trial15342161-44 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:446: finishing write tid 4 to trial15342161-44 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:446: finishing write tid 5 to trial15342161-44 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:446: oid 44 updating version 0 to 78 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:446: oid 44 version 78 is already newer than 78 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:446: oid 44 version 78 is already newer than 78 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:446: oid 44 version 78 is already newer than 78 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 44 v 78 (ObjNum 226 snap 58 seq_num 226) dirty exists 2026-02-20T12:01:18.907 INFO:tasks.rados.rados.0.trial153.stdout:446: left oid 44 (ObjNum 226 snap 58 seq_num 226) 2026-02-20T12:01:19.216 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:19.289 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:19.291 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:19.292 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:19.514 INFO:tasks.rados.rados.0.trial153.stdout:444: done (4 left) 2026-02-20T12:01:19.515 INFO:tasks.rados.rados.0.trial153.stdout:445: done (3 left) 2026-02-20T12:01:19.515 INFO:tasks.rados.rados.0.trial153.stdout:446: done (2 left) 2026-02-20T12:01:19.515 INFO:tasks.rados.rados.0.trial153.stdout:447: done (1 left) 2026-02-20T12:01:19.515 INFO:tasks.rados.rados.0.trial153.stdout:448: done (0 left) 2026-02-20T12:01:19.515 INFO:tasks.rados.rados.0.trial153.stdout:449: snap_remove snap 25 2026-02-20T12:01:20.106 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:20.182 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:20.176+0000 7f348f9a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:20.184 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:20.179+0000 7f348f9a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:20.210 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:20.231 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:20.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~127s 2026-02-20T12:01:20.518 INFO:tasks.rados.rados.0.trial153.stdout:449: done (0 left) 2026-02-20T12:01:20.518 INFO:tasks.rados.rados.0.trial153.stdout:450: write (excl) oid 30 current snap is 59 2026-02-20T12:01:20.518 INFO:tasks.rados.rados.0.trial153.stdout:450: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2026-02-20T12:01:20.522 INFO:tasks.rados.rados.0.trial153.stdout:450: writing trial15342161-30 from 615189 to 1070949 tid 1 2026-02-20T12:01:20.527 INFO:tasks.rados.rados.0.trial153.stdout:450: writing trial15342161-30 from 1733883 to 2444961 tid 2 2026-02-20T12:01:20.532 INFO:tasks.rados.rados.0.trial153.stdout:450: writing trial15342161-30 from 3134431 to 3594472 tid 3 2026-02-20T12:01:20.532 INFO:tasks.rados.rados.0.trial153.stdout:451: snap_create 2026-02-20T12:01:20.562 INFO:tasks.rados.rados.0.trial153.stdout:450: finishing write tid 1 to trial15342161-30 2026-02-20T12:01:20.595 INFO:tasks.rados.rados.0.trial153.stdout:450: finishing write tid 2 to trial15342161-30 2026-02-20T12:01:20.602 INFO:tasks.rados.rados.0.trial153.stdout:450: finishing write tid 3 to trial15342161-30 2026-02-20T12:01:20.613 INFO:tasks.rados.rados.0.trial153.stdout:450: finishing write tid 4 to trial15342161-30 2026-02-20T12:01:20.617 INFO:tasks.rados.rados.0.trial153.stdout:450: finishing write tid 5 to trial15342161-30 2026-02-20T12:01:20.617 INFO:tasks.rados.rados.0.trial153.stdout:450: oid 30 updating version 0 to 79 2026-02-20T12:01:20.617 INFO:tasks.rados.rados.0.trial153.stdout:450: oid 30 updating version 79 to 82 2026-02-20T12:01:20.617 INFO:tasks.rados.rados.0.trial153.stdout:450: oid 30 version 82 is already newer than 82 2026-02-20T12:01:20.617 INFO:tasks.rados.rados.0.trial153.stdout:450: oid 30 version 82 is already newer than 82 2026-02-20T12:01:20.617 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 30 v 82 (ObjNum 227 snap 59 seq_num 227) dirty exists 2026-02-20T12:01:20.618 INFO:tasks.rados.rados.0.trial153.stdout:450: left oid 30 (ObjNum 227 snap 59 seq_num 227) 2026-02-20T12:01:21.520 INFO:tasks.rados.rados.0.trial153.stdout:450: done (1 left) 2026-02-20T12:01:21.520 INFO:tasks.rados.rados.0.trial153.stdout:451: done (0 left) 2026-02-20T12:01:21.520 INFO:tasks.rados.rados.0.trial153.stdout:452: write (excl) oid 2 current snap is 60 2026-02-20T12:01:21.520 INFO:tasks.rados.rados.0.trial153.stdout:452: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2026-02-20T12:01:21.524 INFO:tasks.rados.rados.0.trial153.stdout:452: writing trial15342161-2 from 631996 to 1163005 tid 1 2026-02-20T12:01:21.528 INFO:tasks.rados.rados.0.trial153.stdout:452: writing trial15342161-2 from 1792365 to 2247101 tid 2 2026-02-20T12:01:21.533 INFO:tasks.rados.rados.0.trial153.stdout:452: writing trial15342161-2 from 2761854 to 3433167 tid 3 2026-02-20T12:01:21.533 INFO:tasks.rados.rados.0.trial153.stdout:453: read oid 35 snap -1 2026-02-20T12:01:21.533 INFO:tasks.rados.rados.0.trial153.stdout:453: expect (ObjNum 220 snap 56 seq_num 220) 2026-02-20T12:01:21.533 INFO:tasks.rados.rados.0.trial153.stdout:454: snap_remove snap 52 2026-02-20T12:01:22.231 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:22.310 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:22.304+0000 7f38fc1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:22.312 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:22.308+0000 7f38fc1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:22.338 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:22.355 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:22.524 INFO:tasks.rados.rados.0.trial153.stdout:454: done (2 left) 2026-02-20T12:01:22.524 INFO:tasks.rados.rados.0.trial153.stdout:455: write (excl) oid 9 current snap is 60 2026-02-20T12:01:22.524 INFO:tasks.rados.rados.0.trial153.stdout:455: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2026-02-20T12:01:22.528 INFO:tasks.rados.rados.0.trial153.stdout:455: writing trial15342161-9 from 648803 to 1371414 tid 1 2026-02-20T12:01:22.534 INFO:tasks.rados.rados.0.trial153.stdout:455: writing trial15342161-9 from 1967200 to 2681947 tid 2 2026-02-20T12:01:22.539 INFO:tasks.rados.rados.0.trial153.stdout:455: writing trial15342161-9 from 3421983 to 3848803 tid 3 2026-02-20T12:01:22.539 INFO:tasks.rados.rados.0.trial153.stdout:456: read oid 47 snap -1 2026-02-20T12:01:22.539 INFO:tasks.rados.rados.0.trial153.stdout:456: expect (ObjNum 195 snap 47 seq_num 195) 2026-02-20T12:01:22.539 INFO:tasks.rados.rados.0.trial153.stdout:457: read oid 41 snap -1 2026-02-20T12:01:22.539 INFO:tasks.rados.rados.0.trial153.stdout:457: expect (ObjNum 152 snap 27 seq_num 152) 2026-02-20T12:01:22.539 INFO:tasks.rados.rados.0.trial153.stdout:458: write oid 3 current snap is 60 2026-02-20T12:01:22.539 INFO:tasks.rados.rados.0.trial153.stdout:458: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2026-02-20T12:01:22.544 INFO:tasks.rados.rados.0.trial153.stdout:458: writing trial15342161-3 from 665610 to 1463470 tid 1 2026-02-20T12:01:22.547 INFO:tasks.rados.rados.0.trial153.stdout:458: writing trial15342161-3 from 1909329 to 2367734 tid 2 2026-02-20T12:01:22.551 INFO:tasks.rados.rados.0.trial153.stdout:458: writing trial15342161-3 from 2816700 to 3226910 tid 3 2026-02-20T12:01:22.551 INFO:tasks.rados.rados.0.trial153.stdout:459: write (excl) oid 7 current snap is 60 2026-02-20T12:01:22.551 INFO:tasks.rados.rados.0.trial153.stdout:453: expect (ObjNum 220 snap 56 seq_num 220) 2026-02-20T12:01:22.640 INFO:tasks.rados.rados.0.trial153.stdout:452: finishing write tid 1 to trial15342161-2 2026-02-20T12:01:22.640 INFO:tasks.rados.rados.0.trial153.stdout:452: finishing write tid 2 to trial15342161-2 2026-02-20T12:01:22.641 INFO:tasks.rados.rados.0.trial153.stdout:459: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2026-02-20T12:01:22.644 INFO:tasks.rados.rados.0.trial153.stdout:459: writing trial15342161-7 from 682417 to 1155526 tid 1 2026-02-20T12:01:22.648 INFO:tasks.rados.rados.0.trial153.stdout:459: writing trial15342161-7 from 1567811 to 2169874 tid 2 2026-02-20T12:01:22.653 INFO:tasks.rados.rados.0.trial153.stdout:459: writing trial15342161-7 from 2844123 to 3465605 tid 3 2026-02-20T12:01:22.653 INFO:tasks.rados.rados.0.trial153.stdout:453: done (6 left) 2026-02-20T12:01:22.653 INFO:tasks.rados.rados.0.trial153.stdout:460: write (excl) oid 39 current snap is 60 2026-02-20T12:01:22.653 INFO:tasks.rados.rados.0.trial153.stdout:452: finishing write tid 3 to trial15342161-2 2026-02-20T12:01:22.653 INFO:tasks.rados.rados.0.trial153.stdout:460: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2026-02-20T12:01:22.658 INFO:tasks.rados.rados.0.trial153.stdout:460: writing trial15342161-39 from 699224 to 1247582 tid 1 2026-02-20T12:01:22.662 INFO:tasks.rados.rados.0.trial153.stdout:460: writing trial15342161-39 from 2026293 to 2488367 tid 2 2026-02-20T12:01:22.668 INFO:tasks.rados.rados.0.trial153.stdout:460: writing trial15342161-39 from 3271546 to 3704300 tid 3 2026-02-20T12:01:22.668 INFO:tasks.rados.rados.0.trial153.stdout:461: read oid 29 snap -1 2026-02-20T12:01:22.668 INFO:tasks.rados.rados.0.trial153.stdout:461: expect (ObjNum 148 snap 26 seq_num 148) 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:462: read oid 11 snap 47 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:462: expect (ObjNum 199 snap 47 seq_num 199) 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:463: read oid 13 snap -1 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:463: expect deleted 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:464: read oid 33 snap 48 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:464: expect (ObjNum 165 snap 35 seq_num 165) 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:465: read oid 5 snap 56 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:465: expect deleted 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:466: delete oid 23 current snap is 60 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:452: finishing write tid 4 to trial15342161-2 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:452: finishing write tid 5 to trial15342161-2 2026-02-20T12:01:22.669 INFO:tasks.rados.rados.0.trial153.stdout:452: oid 2 updating version 0 to 174 2026-02-20T12:01:22.670 INFO:tasks.rados.rados.0.trial153.stdout:452: oid 2 version 174 is already newer than 171 2026-02-20T12:01:22.670 INFO:tasks.rados.rados.0.trial153.stdout:452: oid 2 version 174 is already newer than 174 2026-02-20T12:01:22.670 INFO:tasks.rados.rados.0.trial153.stdout:452: oid 2 version 174 is already newer than 174 2026-02-20T12:01:22.670 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 2 v 174 (ObjNum 228 snap 60 seq_num 228) dirty exists 2026-02-20T12:01:22.670 INFO:tasks.rados.rados.0.trial153.stdout:452: left oid 2 (ObjNum 228 snap 60 seq_num 228) 2026-02-20T12:01:22.670 INFO:tasks.rados.rados.0.trial153.stdout:456: expect (ObjNum 195 snap 47 seq_num 195) 2026-02-20T12:01:22.670 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:22.727 INFO:tasks.rados.rados.0.trial153.stdout:457: expect (ObjNum 152 snap 27 seq_num 152) 2026-02-20T12:01:22.746 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1639063 2026-02-20T12:01:22.756 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 750197 2026-02-20T12:01:22.772 INFO:tasks.rados.rados.0.trial153.stdout:455: finishing write tid 1 to trial15342161-9 2026-02-20T12:01:22.772 INFO:tasks.rados.rados.0.trial153.stdout:458: finishing write tid 1 to trial15342161-3 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:458: finishing write tid 2 to trial15342161-3 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:458: finishing write tid 3 to trial15342161-3 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:458: finishing write tid 4 to trial15342161-3 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:455: finishing write tid 2 to trial15342161-9 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:458: finishing write tid 5 to trial15342161-3 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:458: oid 3 updating version 0 to 83 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:458: oid 3 version 83 is already newer than 83 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:458: oid 3 version 83 is already newer than 83 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:458: oid 3 version 83 is already newer than 83 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 3 v 83 (ObjNum 230 snap 60 seq_num 230) dirty exists 2026-02-20T12:01:22.773 INFO:tasks.rados.rados.0.trial153.stdout:458: left oid 3 (ObjNum 230 snap 60 seq_num 230) 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:455: finishing write tid 3 to trial15342161-9 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:455: finishing write tid 4 to trial15342161-9 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:452: done (12 left) 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:456: done (11 left) 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:457: done (10 left) 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:458: done (9 left) 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:466: done (8 left) 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:467: write oid 50 current snap is 60 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:455: finishing write tid 5 to trial15342161-9 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:455: oid 9 updating version 0 to 176 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:455: oid 9 updating version 176 to 179 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:455: oid 9 version 179 is already newer than 179 2026-02-20T12:01:22.774 INFO:tasks.rados.rados.0.trial153.stdout:455: oid 9 version 179 is already newer than 179 2026-02-20T12:01:22.775 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 9 v 179 (ObjNum 229 snap 60 seq_num 229) dirty exists 2026-02-20T12:01:22.775 INFO:tasks.rados.rados.0.trial153.stdout:455: left oid 9 (ObjNum 229 snap 60 seq_num 229) 2026-02-20T12:01:22.775 INFO:tasks.rados.rados.0.trial153.stdout:467: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2026-02-20T12:01:22.777 INFO:tasks.rados.rados.0.trial153.stdout:467: writing trial15342161-50 from 732838 to 1431694 tid 1 2026-02-20T12:01:22.781 INFO:tasks.rados.rados.0.trial153.stdout:467: writing trial15342161-50 from 2026904 to 2492647 tid 2 2026-02-20T12:01:22.786 INFO:tasks.rados.rados.0.trial153.stdout:467: writing trial15342161-50 from 3210039 to 3781690 tid 3 2026-02-20T12:01:22.787 INFO:tasks.rados.rados.0.trial153.stdout:455: done (8 left) 2026-02-20T12:01:22.787 INFO:tasks.rados.rados.0.trial153.stdout:459: finishing write tid 1 to trial15342161-7 2026-02-20T12:01:22.787 INFO:tasks.rados.rados.0.trial153.stdout:468: read oid 43 snap -1 2026-02-20T12:01:22.787 INFO:tasks.rados.rados.0.trial153.stdout:468: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:01:22.787 INFO:tasks.rados.rados.0.trial153.stdout:469: snap_remove snap 59 2026-02-20T12:01:23.528 INFO:tasks.rados.rados.0.trial153.stdout:469: done (9 left) 2026-02-20T12:01:23.528 INFO:tasks.rados.rados.0.trial153.stdout:470: write (excl) oid 34 current snap is 60 2026-02-20T12:01:23.529 INFO:tasks.rados.rados.0.trial153.stdout:470: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2026-02-20T12:01:23.534 INFO:tasks.rados.rados.0.trial153.stdout:470: writing trial15342161-34 from 749645 to 1523750 tid 1 2026-02-20T12:01:23.539 INFO:tasks.rados.rados.0.trial153.stdout:470: writing trial15342161-34 from 2085386 to 2694787 tid 2 2026-02-20T12:01:23.544 INFO:tasks.rados.rados.0.trial153.stdout:470: writing trial15342161-34 from 3237462 to 3949645 tid 3 2026-02-20T12:01:23.544 INFO:tasks.rados.rados.0.trial153.stdout:471: write oid 19 current snap is 60 2026-02-20T12:01:23.544 INFO:tasks.rados.rados.0.trial153.stdout:471: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2026-02-20T12:01:23.548 INFO:tasks.rados.rados.0.trial153.stdout:471: writing trial15342161-19 from 766452 to 1332159 tid 1 2026-02-20T12:01:23.552 INFO:tasks.rados.rados.0.trial153.stdout:471: writing trial15342161-19 from 1860221 to 2329633 tid 2 2026-02-20T12:01:23.558 INFO:tasks.rados.rados.0.trial153.stdout:471: writing trial15342161-19 from 2981238 to 3575433 tid 3 2026-02-20T12:01:23.558 INFO:tasks.rados.rados.0.trial153.stdout:472: read oid 30 snap -1 2026-02-20T12:01:23.558 INFO:tasks.rados.rados.0.trial153.stdout:472: expect (ObjNum 227 snap 59 seq_num 227) 2026-02-20T12:01:23.558 INFO:tasks.rados.rados.0.trial153.stdout:473: read oid 26 snap 47 2026-02-20T12:01:23.558 INFO:tasks.rados.rados.0.trial153.stdout:473: expect (ObjNum 147 snap 25 seq_num 147) 2026-02-20T12:01:23.558 INFO:tasks.rados.rados.0.trial153.stdout:474: read oid 6 snap -1 2026-02-20T12:01:23.558 INFO:tasks.rados.rados.0.trial153.stdout:474: expect deleted 2026-02-20T12:01:23.558 INFO:tasks.rados.rados.0.trial153.stdout:465: done (13 left) 2026-02-20T12:01:23.558 INFO:tasks.rados.rados.0.trial153.stdout:475: write oid 18 current snap is 60 2026-02-20T12:01:23.559 INFO:tasks.rados.rados.0.trial153.stdout:475: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2026-02-20T12:01:23.562 INFO:tasks.rados.rados.0.trial153.stdout:475: writing trial15342161-18 from 783259 to 1424215 tid 1 2026-02-20T12:01:23.567 INFO:tasks.rados.rados.0.trial153.stdout:475: writing trial15342161-18 from 1918703 to 2531773 tid 2 2026-02-20T12:01:23.570 INFO:tasks.rados.rados.0.trial153.stdout:475: writing trial15342161-18 from 3008661 to 3414128 tid 3 2026-02-20T12:01:23.572 INFO:tasks.rados.rados.0.trial153.stdout:475: writing trial15342161-18 from 3821939 to 3983259 tid 4 2026-02-20T12:01:23.572 INFO:tasks.rados.rados.0.trial153.stdout:476: delete oid 45 current snap is 60 2026-02-20T12:01:23.573 INFO:tasks.rados.rados.0.trial153.stdout:462: expect (ObjNum 199 snap 47 seq_num 199) 2026-02-20T12:01:23.580 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2199285 2026-02-20T12:01:23.589 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:23.617 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:23.645 INFO:tasks.rados.rados.0.trial153.stdout:461: expect (ObjNum 148 snap 26 seq_num 148) 2026-02-20T12:01:23.675 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:23.726 INFO:tasks.rados.rados.0.trial153.stdout:459: finishing write tid 2 to trial15342161-7 2026-02-20T12:01:23.726 INFO:tasks.rados.rados.0.trial153.stdout:461: done (14 left) 2026-02-20T12:01:23.726 INFO:tasks.rados.rados.0.trial153.stdout:462: done (13 left) 2026-02-20T12:01:23.726 INFO:tasks.rados.rados.0.trial153.stdout:463: done (12 left) 2026-02-20T12:01:23.726 INFO:tasks.rados.rados.0.trial153.stdout:476: done (11 left) 2026-02-20T12:01:23.726 INFO:tasks.rados.rados.0.trial153.stdout:477: read oid 27 snap -1 2026-02-20T12:01:23.726 INFO:tasks.rados.rados.0.trial153.stdout:477: expect (ObjNum 201 snap 50 seq_num 201) 2026-02-20T12:01:23.727 INFO:tasks.rados.rados.0.trial153.stdout:478: read oid 4 snap -1 2026-02-20T12:01:23.727 INFO:tasks.rados.rados.0.trial153.stdout:478: expect (ObjNum 222 snap 56 seq_num 222) 2026-02-20T12:01:23.727 INFO:tasks.rados.rados.0.trial153.stdout:459: finishing write tid 3 to trial15342161-7 2026-02-20T12:01:23.727 INFO:tasks.rados.rados.0.trial153.stdout:464: expect (ObjNum 165 snap 35 seq_num 165) 2026-02-20T12:01:23.784 INFO:tasks.rados.rados.0.trial153.stdout:459: finishing write tid 4 to trial15342161-7 2026-02-20T12:01:23.784 INFO:tasks.rados.rados.0.trial153.stdout:459: finishing write tid 5 to trial15342161-7 2026-02-20T12:01:23.784 INFO:tasks.rados.rados.0.trial153.stdout:459: oid 7 updating version 0 to 57 2026-02-20T12:01:23.784 INFO:tasks.rados.rados.0.trial153.stdout:459: oid 7 version 57 is already newer than 57 2026-02-20T12:01:23.784 INFO:tasks.rados.rados.0.trial153.stdout:459: oid 7 version 57 is already newer than 54 2026-02-20T12:01:23.784 INFO:tasks.rados.rados.0.trial153.stdout:459: oid 7 version 57 is already newer than 57 2026-02-20T12:01:23.784 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 7 v 57 (ObjNum 231 snap 60 seq_num 231) dirty exists 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:459: left oid 7 (ObjNum 231 snap 60 seq_num 231) 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:460: finishing write tid 1 to trial15342161-39 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:460: finishing write tid 2 to trial15342161-39 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:459: done (12 left) 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:464: done (11 left) 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:479: write (excl) oid 16 current snap is 60 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:460: finishing write tid 3 to trial15342161-39 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:460: finishing write tid 4 to trial15342161-39 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:460: finishing write tid 5 to trial15342161-39 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:460: oid 39 updating version 0 to 125 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:460: oid 39 version 125 is already newer than 125 2026-02-20T12:01:23.785 INFO:tasks.rados.rados.0.trial153.stdout:460: oid 39 version 125 is already newer than 125 2026-02-20T12:01:23.786 INFO:tasks.rados.rados.0.trial153.stdout:460: oid 39 version 125 is already newer than 122 2026-02-20T12:01:23.786 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 39 v 125 (ObjNum 232 snap 60 seq_num 232) dirty exists 2026-02-20T12:01:23.786 INFO:tasks.rados.rados.0.trial153.stdout:460: left oid 39 (ObjNum 232 snap 60 seq_num 232) 2026-02-20T12:01:23.786 INFO:tasks.rados.rados.0.trial153.stdout:479: seq_num 239 ranges {416873=791454,1919314=97559} 2026-02-20T12:01:23.788 INFO:tasks.rados.rados.0.trial153.stdout:479: writing trial15342161-16 from 416873 to 1208327 tid 1 2026-02-20T12:01:23.791 INFO:tasks.rados.rados.0.trial153.stdout:479: writing trial15342161-16 from 1919314 to 2016873 tid 2 2026-02-20T12:01:23.791 INFO:tasks.rados.rados.0.trial153.stdout:460: done (11 left) 2026-02-20T12:01:23.791 INFO:tasks.rados.rados.0.trial153.stdout:480: delete oid 41 current snap is 60 2026-02-20T12:01:23.791 INFO:tasks.rados.rados.0.trial153.stdout:468: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:01:23.801 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1259262 2026-02-20T12:01:23.907 INFO:tasks.rados.rados.0.trial153.stdout:467: finishing write tid 1 to trial15342161-50 2026-02-20T12:01:23.907 INFO:tasks.rados.rados.0.trial153.stdout:467: finishing write tid 2 to trial15342161-50 2026-02-20T12:01:23.908 INFO:tasks.rados.rados.0.trial153.stdout:468: done (11 left) 2026-02-20T12:01:23.908 INFO:tasks.rados.rados.0.trial153.stdout:480: done (10 left) 2026-02-20T12:01:23.908 INFO:tasks.rados.rados.0.trial153.stdout:481: read oid 46 snap 50 2026-02-20T12:01:23.908 INFO:tasks.rados.rados.0.trial153.stdout:481: expect deleted 2026-02-20T12:01:23.908 INFO:tasks.rados.rados.0.trial153.stdout:482: snap_remove snap 58 2026-02-20T12:01:24.293 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:24.356 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:24.366 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:24.369 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:24.434 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:24.428+0000 7fabb9374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:24.437 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:24.432+0000 7fabb9374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:24.463 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:24.479 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:24.530 INFO:tasks.rados.rados.0.trial153.stdout:482: done (11 left) 2026-02-20T12:01:24.531 INFO:tasks.rados.rados.0.trial153.stdout:483: snap_remove snap 56 2026-02-20T12:01:25.533 INFO:tasks.rados.rados.0.trial153.stdout:483: done (11 left) 2026-02-20T12:01:25.534 INFO:tasks.rados.rados.0.trial153.stdout:484: read oid 10 snap -1 2026-02-20T12:01:25.534 INFO:tasks.rados.rados.0.trial153.stdout:484: expect (ObjNum 218 snap 56 seq_num 218) 2026-02-20T12:01:25.534 INFO:tasks.rados.rados.0.trial153.stdout:485: write oid 5 current snap is 60 2026-02-20T12:01:25.534 INFO:tasks.rados.rados.0.trial153.stdout:467: finishing write tid 3 to trial15342161-50 2026-02-20T12:01:25.534 INFO:tasks.rados.rados.0.trial153.stdout:485: seq_num 241 ranges {450487=541952,1636278=414209} 2026-02-20T12:01:25.537 INFO:tasks.rados.rados.0.trial153.stdout:485: writing trial15342161-5 from 450487 to 992439 tid 1 2026-02-20T12:01:25.541 INFO:tasks.rados.rados.0.trial153.stdout:485: writing trial15342161-5 from 1636278 to 2050487 tid 2 2026-02-20T12:01:25.541 INFO:tasks.rados.rados.0.trial153.stdout:486: delete oid 21 current snap is 60 2026-02-20T12:01:25.542 INFO:tasks.rados.rados.0.trial153.stdout:467: finishing write tid 4 to trial15342161-50 2026-02-20T12:01:25.542 INFO:tasks.rados.rados.0.trial153.stdout:467: finishing write tid 5 to trial15342161-50 2026-02-20T12:01:25.542 INFO:tasks.rados.rados.0.trial153.stdout:467: oid 50 updating version 0 to 105 2026-02-20T12:01:25.542 INFO:tasks.rados.rados.0.trial153.stdout:467: oid 50 version 105 is already newer than 105 2026-02-20T12:01:25.542 INFO:tasks.rados.rados.0.trial153.stdout:467: oid 50 version 105 is already newer than 105 2026-02-20T12:01:25.542 INFO:tasks.rados.rados.0.trial153.stdout:467: oid 50 version 105 is already newer than 105 2026-02-20T12:01:25.542 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 50 v 105 (ObjNum 234 snap 60 seq_num 234) dirty exists 2026-02-20T12:01:25.542 INFO:tasks.rados.rados.0.trial153.stdout:467: left oid 50 (ObjNum 234 snap 60 seq_num 234) 2026-02-20T12:01:25.542 INFO:tasks.rados.rados.0.trial153.stdout:473: expect (ObjNum 147 snap 25 seq_num 147) 2026-02-20T12:01:25.621 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 94121 2026-02-20T12:01:25.667 INFO:tasks.rados.rados.0.trial153.stdout:470: finishing write tid 1 to trial15342161-34 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:470: finishing write tid 2 to trial15342161-34 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:467: done (13 left) 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:473: done (12 left) 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:474: done (11 left) 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:486: done (10 left) 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:487: read oid 37 snap -1 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:487: expect deleted 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:475: finishing write tid 1 to trial15342161-18 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:470: finishing write tid 3 to trial15342161-34 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:475: finishing write tid 2 to trial15342161-18 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:488: read oid 31 snap -1 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:488: expect (ObjNum 128 snap 19 seq_num 128) 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:489: read oid 20 snap -1 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:489: expect (ObjNum 126 snap 16 seq_num 126) 2026-02-20T12:01:25.668 INFO:tasks.rados.rados.0.trial153.stdout:490: snap_remove snap 48 2026-02-20T12:01:25.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~132s 2026-02-20T12:01:26.480 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:26.506 INFO:tasks.rados.rados.0.trial153.stdout:490: done (13 left) 2026-02-20T12:01:26.506 INFO:tasks.rados.rados.0.trial153.stdout:491: snap_remove snap 47 2026-02-20T12:01:26.559 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:26.553+0000 7f05d4b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:26.561 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:26.557+0000 7f05d4b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:26.587 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:26.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:27.511 INFO:tasks.rados.rados.0.trial153.stdout:491: done (13 left) 2026-02-20T12:01:27.511 INFO:tasks.rados.rados.0.trial153.stdout:492: read oid 3 snap -1 2026-02-20T12:01:27.511 INFO:tasks.rados.rados.0.trial153.stdout:492: expect (ObjNum 230 snap 60 seq_num 230) 2026-02-20T12:01:27.511 INFO:tasks.rados.rados.0.trial153.stdout:493: delete oid 25 current snap is 60 2026-02-20T12:01:27.511 INFO:tasks.rados.rados.0.trial153.stdout:472: expect (ObjNum 227 snap 59 seq_num 227) 2026-02-20T12:01:27.577 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:27.610 INFO:tasks.rados.rados.0.trial153.stdout:475: finishing write tid 3 to trial15342161-18 2026-02-20T12:01:27.610 INFO:tasks.rados.rados.0.trial153.stdout:472: done (14 left) 2026-02-20T12:01:27.611 INFO:tasks.rados.rados.0.trial153.stdout:493: done (13 left) 2026-02-20T12:01:27.611 INFO:tasks.rados.rados.0.trial153.stdout:494: snap_create 2026-02-20T12:01:27.611 INFO:tasks.rados.rados.0.trial153.stdout:470: finishing write tid 4 to trial15342161-34 2026-02-20T12:01:27.611 INFO:tasks.rados.rados.0.trial153.stdout:470: finishing write tid 5 to trial15342161-34 2026-02-20T12:01:27.611 INFO:tasks.rados.rados.0.trial153.stdout:470: oid 34 updating version 0 to 87 2026-02-20T12:01:27.611 INFO:tasks.rados.rados.0.trial153.stdout:470: oid 34 version 87 is already newer than 87 2026-02-20T12:01:27.611 INFO:tasks.rados.rados.0.trial153.stdout:470: oid 34 version 87 is already newer than 84 2026-02-20T12:01:27.612 INFO:tasks.rados.rados.0.trial153.stdout:470: oid 34 version 87 is already newer than 87 2026-02-20T12:01:27.612 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 87 (ObjNum 235 snap 60 seq_num 235) dirty exists 2026-02-20T12:01:27.612 INFO:tasks.rados.rados.0.trial153.stdout:470: left oid 34 (ObjNum 235 snap 60 seq_num 235) 2026-02-20T12:01:27.612 INFO:tasks.rados.rados.0.trial153.stdout:475: finishing write tid 4 to trial15342161-18 2026-02-20T12:01:27.612 INFO:tasks.rados.rados.0.trial153.stdout:475: finishing write tid 5 to trial15342161-18 2026-02-20T12:01:27.612 INFO:tasks.rados.rados.0.trial153.stdout:475: finishing write tid 6 to trial15342161-18 2026-02-20T12:01:27.612 INFO:tasks.rados.rados.0.trial153.stdout:475: oid 18 updating version 0 to 29 2026-02-20T12:01:27.613 INFO:tasks.rados.rados.0.trial153.stdout:475: oid 18 version 29 is already newer than 29 2026-02-20T12:01:27.613 INFO:tasks.rados.rados.0.trial153.stdout:475: oid 18 version 29 is already newer than 29 2026-02-20T12:01:27.613 INFO:tasks.rados.rados.0.trial153.stdout:475: oid 18 version 29 is already newer than 29 2026-02-20T12:01:27.613 INFO:tasks.rados.rados.0.trial153.stdout:475: oid 18 version 29 is already newer than 29 2026-02-20T12:01:27.613 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 18 v 29 (ObjNum 237 snap 60 seq_num 237) dirty exists 2026-02-20T12:01:27.613 INFO:tasks.rados.rados.0.trial153.stdout:475: left oid 18 (ObjNum 237 snap 60 seq_num 237) 2026-02-20T12:01:27.613 INFO:tasks.rados.rados.0.trial153.stdout:471: finishing write tid 1 to trial15342161-19 2026-02-20T12:01:27.613 INFO:tasks.rados.rados.0.trial153.stdout:471: finishing write tid 2 to trial15342161-19 2026-02-20T12:01:27.613 INFO:tasks.rados.rados.0.trial153.stdout:471: finishing write tid 3 to trial15342161-19 2026-02-20T12:01:27.614 INFO:tasks.rados.rados.0.trial153.stdout:471: finishing write tid 4 to trial15342161-19 2026-02-20T12:01:27.614 INFO:tasks.rados.rados.0.trial153.stdout:471: finishing write tid 5 to trial15342161-19 2026-02-20T12:01:27.614 INFO:tasks.rados.rados.0.trial153.stdout:471: oid 19 updating version 0 to 110 2026-02-20T12:01:27.614 INFO:tasks.rados.rados.0.trial153.stdout:471: oid 19 version 110 is already newer than 110 2026-02-20T12:01:27.614 INFO:tasks.rados.rados.0.trial153.stdout:471: oid 19 version 110 is already newer than 110 2026-02-20T12:01:27.614 INFO:tasks.rados.rados.0.trial153.stdout:471: oid 19 version 110 is already newer than 110 2026-02-20T12:01:27.614 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 110 (ObjNum 236 snap 60 seq_num 236) dirty exists 2026-02-20T12:01:27.614 INFO:tasks.rados.rados.0.trial153.stdout:471: left oid 19 (ObjNum 236 snap 60 seq_num 236) 2026-02-20T12:01:27.614 INFO:tasks.rados.rados.0.trial153.stdout:477: expect (ObjNum 201 snap 50 seq_num 201) 2026-02-20T12:01:27.678 INFO:tasks.rados.rados.0.trial153.stdout:478: expect (ObjNum 222 snap 56 seq_num 222) 2026-02-20T12:01:27.815 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1145420 2026-02-20T12:01:27.872 INFO:tasks.rados.rados.0.trial153.stdout:479: finishing write tid 1 to trial15342161-16 2026-02-20T12:01:27.873 INFO:tasks.rados.rados.0.trial153.stdout:479: finishing write tid 2 to trial15342161-16 2026-02-20T12:01:27.873 INFO:tasks.rados.rados.0.trial153.stdout:479: finishing write tid 3 to trial15342161-16 2026-02-20T12:01:27.873 INFO:tasks.rados.rados.0.trial153.stdout:479: finishing write tid 4 to trial15342161-16 2026-02-20T12:01:27.873 INFO:tasks.rados.rados.0.trial153.stdout:479: oid 16 updating version 0 to 91 2026-02-20T12:01:27.873 INFO:tasks.rados.rados.0.trial153.stdout:479: oid 16 version 91 is already newer than 91 2026-02-20T12:01:27.873 INFO:tasks.rados.rados.0.trial153.stdout:479: oid 16 version 91 is already newer than 89 2026-02-20T12:01:27.873 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 16 v 91 (ObjNum 239 snap 60 seq_num 239) dirty exists 2026-02-20T12:01:27.873 INFO:tasks.rados.rados.0.trial153.stdout:479: left oid 16 (ObjNum 239 snap 60 seq_num 239) 2026-02-20T12:01:27.873 INFO:tasks.rados.rados.0.trial153.stdout:485: finishing write tid 1 to trial15342161-5 2026-02-20T12:01:27.874 INFO:tasks.rados.rados.0.trial153.stdout:485: finishing write tid 2 to trial15342161-5 2026-02-20T12:01:27.874 INFO:tasks.rados.rados.0.trial153.stdout:484: expect (ObjNum 218 snap 56 seq_num 218) 2026-02-20T12:01:27.882 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1794320 2026-02-20T12:01:28.032 INFO:tasks.rados.rados.0.trial153.stdout:485: finishing write tid 3 to trial15342161-5 2026-02-20T12:01:28.033 INFO:tasks.rados.rados.0.trial153.stdout:485: finishing write tid 4 to trial15342161-5 2026-02-20T12:01:28.033 INFO:tasks.rados.rados.0.trial153.stdout:485: oid 5 updating version 0 to 52 2026-02-20T12:01:28.033 INFO:tasks.rados.rados.0.trial153.stdout:485: oid 5 version 52 is already newer than 52 2026-02-20T12:01:28.033 INFO:tasks.rados.rados.0.trial153.stdout:485: oid 5 version 52 is already newer than 52 2026-02-20T12:01:28.033 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 5 v 52 (ObjNum 241 snap 60 seq_num 241) dirty exists 2026-02-20T12:01:28.033 INFO:tasks.rados.rados.0.trial153.stdout:485: left oid 5 (ObjNum 241 snap 60 seq_num 241) 2026-02-20T12:01:28.033 INFO:tasks.rados.rados.0.trial153.stdout:488: expect (ObjNum 128 snap 19 seq_num 128) 2026-02-20T12:01:28.037 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 897441 2026-02-20T12:01:28.062 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1726806 2026-02-20T12:01:28.076 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1158047 2026-02-20T12:01:28.090 INFO:tasks.rados.rados.0.trial153.stdout:489: expect (ObjNum 126 snap 16 seq_num 126) 2026-02-20T12:01:28.090 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:28.136 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:28.158 INFO:tasks.rados.rados.0.trial153.stdout:492: expect (ObjNum 230 snap 60 seq_num 230) 2026-02-20T12:01:28.284 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:28.540 INFO:tasks.rados.rados.0.trial153.stdout:470: done (13 left) 2026-02-20T12:01:28.540 INFO:tasks.rados.rados.0.trial153.stdout:471: done (12 left) 2026-02-20T12:01:28.540 INFO:tasks.rados.rados.0.trial153.stdout:475: done (11 left) 2026-02-20T12:01:28.540 INFO:tasks.rados.rados.0.trial153.stdout:477: done (10 left) 2026-02-20T12:01:28.540 INFO:tasks.rados.rados.0.trial153.stdout:478: done (9 left) 2026-02-20T12:01:28.540 INFO:tasks.rados.rados.0.trial153.stdout:479: done (8 left) 2026-02-20T12:01:28.540 INFO:tasks.rados.rados.0.trial153.stdout:481: done (7 left) 2026-02-20T12:01:28.540 INFO:tasks.rados.rados.0.trial153.stdout:484: done (6 left) 2026-02-20T12:01:28.540 INFO:tasks.rados.rados.0.trial153.stdout:485: done (5 left) 2026-02-20T12:01:28.541 INFO:tasks.rados.rados.0.trial153.stdout:487: done (4 left) 2026-02-20T12:01:28.541 INFO:tasks.rados.rados.0.trial153.stdout:488: done (3 left) 2026-02-20T12:01:28.541 INFO:tasks.rados.rados.0.trial153.stdout:489: done (2 left) 2026-02-20T12:01:28.541 INFO:tasks.rados.rados.0.trial153.stdout:492: done (1 left) 2026-02-20T12:01:28.541 INFO:tasks.rados.rados.0.trial153.stdout:494: done (0 left) 2026-02-20T12:01:28.541 INFO:tasks.rados.rados.0.trial153.stdout:495: delete oid 6 current snap is 61 2026-02-20T12:01:28.569 INFO:tasks.rados.rados.0.trial153.stdout:495: done (0 left) 2026-02-20T12:01:28.569 INFO:tasks.rados.rados.0.trial153.stdout:496: write (excl) oid 20 current snap is 61 2026-02-20T12:01:28.569 INFO:tasks.rados.rados.0.trial153.stdout:496: seq_num 245 ranges {517715=559301,1870206=247509} 2026-02-20T12:01:28.573 INFO:tasks.rados.rados.0.trial153.stdout:496: writing trial15342161-20 from 517715 to 1077016 tid 1 2026-02-20T12:01:28.577 INFO:tasks.rados.rados.0.trial153.stdout:496: writing trial15342161-20 from 1870206 to 2117715 tid 2 2026-02-20T12:01:28.577 INFO:tasks.rados.rados.0.trial153.stdout:497: snap_remove snap 50 2026-02-20T12:01:28.604 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:28.681 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:28.675+0000 7fb677c11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:28.683 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:28.679+0000 7fb677c11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:28.728 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:28.751 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:29.370 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:29.442 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:29.444 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:29.445 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:29.544 INFO:tasks.rados.rados.0.trial153.stdout:497: done (1 left) 2026-02-20T12:01:29.544 INFO:tasks.rados.rados.0.trial153.stdout:498: delete oid 36 current snap is 61 2026-02-20T12:01:29.544 INFO:tasks.rados.rados.0.trial153.stdout:496: finishing write tid 1 to trial15342161-20 2026-02-20T12:01:29.544 INFO:tasks.rados.rados.0.trial153.stdout:496: finishing write tid 2 to trial15342161-20 2026-02-20T12:01:29.544 INFO:tasks.rados.rados.0.trial153.stdout:496: finishing write tid 3 to trial15342161-20 2026-02-20T12:01:29.544 INFO:tasks.rados.rados.0.trial153.stdout:496: finishing write tid 4 to trial15342161-20 2026-02-20T12:01:29.545 INFO:tasks.rados.rados.0.trial153.stdout:496: oid 20 updating version 0 to 35 2026-02-20T12:01:29.545 INFO:tasks.rados.rados.0.trial153.stdout:496: oid 20 version 35 is already newer than 33 2026-02-20T12:01:29.545 INFO:tasks.rados.rados.0.trial153.stdout:496: oid 20 version 35 is already newer than 35 2026-02-20T12:01:29.545 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 20 v 35 (ObjNum 245 snap 61 seq_num 245) dirty exists 2026-02-20T12:01:29.545 INFO:tasks.rados.rados.0.trial153.stdout:496: left oid 20 (ObjNum 245 snap 61 seq_num 245) 2026-02-20T12:01:29.560 INFO:tasks.rados.rados.0.trial153.stdout:496: done (1 left) 2026-02-20T12:01:29.560 INFO:tasks.rados.rados.0.trial153.stdout:498: done (0 left) 2026-02-20T12:01:29.560 INFO:tasks.rados.rados.0.trial153.stdout:499: delete oid 1 current snap is 61 2026-02-20T12:01:29.587 INFO:tasks.rados.rados.0.trial153.stdout:499: done (0 left) 2026-02-20T12:01:29.587 INFO:tasks.rados.rados.0.trial153.stdout:500: read oid 17 snap -1 2026-02-20T12:01:29.587 INFO:tasks.rados.rados.0.trial153.stdout:500: expect (ObjNum 191 snap 47 seq_num 191) 2026-02-20T12:01:29.587 INFO:tasks.rados.rados.0.trial153.stdout:501: read oid 43 snap -1 2026-02-20T12:01:29.587 INFO:tasks.rados.rados.0.trial153.stdout:501: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:01:29.587 INFO:tasks.rados.rados.0.trial153.stdout:502: read oid 37 snap 60 2026-02-20T12:01:29.587 INFO:tasks.rados.rados.0.trial153.stdout:502: expect deleted 2026-02-20T12:01:29.587 INFO:tasks.rados.rados.0.trial153.stdout:503: read oid 28 snap -1 2026-02-20T12:01:29.587 INFO:tasks.rados.rados.0.trial153.stdout:503: expect (ObjNum 203 snap 51 seq_num 203) 2026-02-20T12:01:29.587 INFO:tasks.rados.rados.0.trial153.stdout:504: snap_create 2026-02-20T12:01:29.607 INFO:tasks.rados.rados.0.trial153.stdout:503: expect (ObjNum 203 snap 51 seq_num 203) 2026-02-20T12:01:29.608 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 159300 2026-02-20T12:01:29.681 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:29.720 INFO:tasks.rados.rados.0.trial153.stdout:500: expect (ObjNum 191 snap 47 seq_num 191) 2026-02-20T12:01:29.720 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:29.784 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2167360 2026-02-20T12:01:29.855 INFO:tasks.rados.rados.0.trial153.stdout:501: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:01:29.855 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:30.548 INFO:tasks.rados.rados.0.trial153.stdout:500: done (4 left) 2026-02-20T12:01:30.548 INFO:tasks.rados.rados.0.trial153.stdout:501: done (3 left) 2026-02-20T12:01:30.548 INFO:tasks.rados.rados.0.trial153.stdout:502: done (2 left) 2026-02-20T12:01:30.548 INFO:tasks.rados.rados.0.trial153.stdout:503: done (1 left) 2026-02-20T12:01:30.548 INFO:tasks.rados.rados.0.trial153.stdout:504: done (0 left) 2026-02-20T12:01:30.548 INFO:tasks.rados.rados.0.trial153.stdout:505: read oid 9 snap -1 2026-02-20T12:01:30.548 INFO:tasks.rados.rados.0.trial153.stdout:505: expect (ObjNum 229 snap 60 seq_num 229) 2026-02-20T12:01:30.548 INFO:tasks.rados.rados.0.trial153.stdout:506: snap_create 2026-02-20T12:01:30.581 INFO:tasks.rados.rados.0.trial153.stdout:505: expect (ObjNum 229 snap 60 seq_num 229) 2026-02-20T12:01:30.698 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:30.751 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:30.827 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:30.822+0000 7f24b8a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:30.830 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:30.825+0000 7f24b8a1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:30.856 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:30.871 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:31.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~138s 2026-02-20T12:01:31.549 INFO:tasks.rados.rados.0.trial153.stdout:505: done (1 left) 2026-02-20T12:01:31.549 INFO:tasks.rados.rados.0.trial153.stdout:506: done (0 left) 2026-02-20T12:01:31.550 INFO:tasks.rados.rados.0.trial153.stdout:507: write (excl) oid 24 current snap is 63 2026-02-20T12:01:31.550 INFO:tasks.rados.rados.0.trial153.stdout:507: seq_num 248 ranges {568136=785048,1929299=238837} 2026-02-20T12:01:31.554 INFO:tasks.rados.rados.0.trial153.stdout:507: writing trial15342161-24 from 568136 to 1353184 tid 1 2026-02-20T12:01:31.558 INFO:tasks.rados.rados.0.trial153.stdout:507: writing trial15342161-24 from 1929299 to 2168136 tid 2 2026-02-20T12:01:31.558 INFO:tasks.rados.rados.0.trial153.stdout:508: read oid 4 snap -1 2026-02-20T12:01:31.558 INFO:tasks.rados.rados.0.trial153.stdout:508: expect (ObjNum 222 snap 56 seq_num 222) 2026-02-20T12:01:31.558 INFO:tasks.rados.rados.0.trial153.stdout:509: snap_remove snap 60 2026-02-20T12:01:32.555 INFO:tasks.rados.rados.0.trial153.stdout:509: done (2 left) 2026-02-20T12:01:32.555 INFO:tasks.rados.rados.0.trial153.stdout:510: snap_create 2026-02-20T12:01:32.555 INFO:tasks.rados.rados.0.trial153.stdout:507: finishing write tid 1 to trial15342161-24 2026-02-20T12:01:32.555 INFO:tasks.rados.rados.0.trial153.stdout:508: expect (ObjNum 222 snap 56 seq_num 222) 2026-02-20T12:01:32.697 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:32.768 INFO:tasks.rados.rados.0.trial153.stdout:507: finishing write tid 2 to trial15342161-24 2026-02-20T12:01:32.768 INFO:tasks.rados.rados.0.trial153.stdout:507: finishing write tid 3 to trial15342161-24 2026-02-20T12:01:32.768 INFO:tasks.rados.rados.0.trial153.stdout:507: finishing write tid 4 to trial15342161-24 2026-02-20T12:01:32.768 INFO:tasks.rados.rados.0.trial153.stdout:507: oid 24 updating version 0 to 21 2026-02-20T12:01:32.768 INFO:tasks.rados.rados.0.trial153.stdout:507: oid 24 updating version 21 to 23 2026-02-20T12:01:32.768 INFO:tasks.rados.rados.0.trial153.stdout:507: oid 24 version 23 is already newer than 23 2026-02-20T12:01:32.768 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 24 v 23 (ObjNum 248 snap 63 seq_num 248) dirty exists 2026-02-20T12:01:32.768 INFO:tasks.rados.rados.0.trial153.stdout:507: left oid 24 (ObjNum 248 snap 63 seq_num 248) 2026-02-20T12:01:32.872 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:32.947 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:32.942+0000 7fc06b0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:32.950 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:32.945+0000 7fc06b0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:32.976 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:32.992 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:33.560 INFO:tasks.rados.rados.0.trial153.stdout:507: done (2 left) 2026-02-20T12:01:33.560 INFO:tasks.rados.rados.0.trial153.stdout:508: done (1 left) 2026-02-20T12:01:33.560 INFO:tasks.rados.rados.0.trial153.stdout:510: done (0 left) 2026-02-20T12:01:33.560 INFO:tasks.rados.rados.0.trial153.stdout:511: read oid 46 snap -1 2026-02-20T12:01:33.560 INFO:tasks.rados.rados.0.trial153.stdout:511: expect (ObjNum 223 snap 58 seq_num 223) 2026-02-20T12:01:33.560 INFO:tasks.rados.rados.0.trial153.stdout:512: read oid 12 snap 63 2026-02-20T12:01:33.560 INFO:tasks.rados.rados.0.trial153.stdout:512: expect deleted 2026-02-20T12:01:33.560 INFO:tasks.rados.rados.0.trial153.stdout:513: snap_remove snap 62 2026-02-20T12:01:34.445 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:34.515 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:34.519 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:34.562 INFO:tasks.rados.rados.0.trial153.stdout:513: done (2 left) 2026-02-20T12:01:34.562 INFO:tasks.rados.rados.0.trial153.stdout:514: delete oid 18 current snap is 64 2026-02-20T12:01:34.562 INFO:tasks.rados.rados.0.trial153.stdout:511: expect (ObjNum 223 snap 58 seq_num 223) 2026-02-20T12:01:34.656 INFO:tasks.rados.rados.0.trial153.stdout:511: done (2 left) 2026-02-20T12:01:34.657 INFO:tasks.rados.rados.0.trial153.stdout:512: done (1 left) 2026-02-20T12:01:34.657 INFO:tasks.rados.rados.0.trial153.stdout:514: done (0 left) 2026-02-20T12:01:34.657 INFO:tasks.rados.rados.0.trial153.stdout:515: snap_remove snap 61 2026-02-20T12:01:34.992 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:35.070 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:35.064+0000 7f27f9e3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:35.072 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:35.068+0000 7f27f9e3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:35.100 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:35.116 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:35.569 INFO:tasks.rados.rados.0.trial153.stdout:515: done (0 left) 2026-02-20T12:01:35.569 INFO:tasks.rados.rados.0.trial153.stdout:516: snap_create 2026-02-20T12:01:36.510 INFO:tasks.rados.rados.0.trial153.stdout:516: done (0 left) 2026-02-20T12:01:36.511 INFO:tasks.rados.rados.0.trial153.stdout:517: read oid 19 snap -1 2026-02-20T12:01:36.511 INFO:tasks.rados.rados.0.trial153.stdout:517: expect (ObjNum 236 snap 60 seq_num 236) 2026-02-20T12:01:36.511 INFO:tasks.rados.rados.0.trial153.stdout:518: read oid 27 snap -1 2026-02-20T12:01:36.511 INFO:tasks.rados.rados.0.trial153.stdout:518: expect (ObjNum 201 snap 50 seq_num 201) 2026-02-20T12:01:36.511 INFO:tasks.rados.rados.0.trial153.stdout:519: write oid 14 current snap is 65 2026-02-20T12:01:36.511 INFO:tasks.rados.rados.0.trial153.stdout:519: seq_num 250 ranges {601750=535546,1929910=271840} 2026-02-20T12:01:36.514 INFO:tasks.rados.rados.0.trial153.stdout:519: writing trial15342161-14 from 601750 to 1137296 tid 1 2026-02-20T12:01:36.518 INFO:tasks.rados.rados.0.trial153.stdout:519: writing trial15342161-14 from 1929910 to 2201750 tid 2 2026-02-20T12:01:36.518 INFO:tasks.rados.rados.0.trial153.stdout:520: write oid 45 current snap is 65 2026-02-20T12:01:36.518 INFO:tasks.rados.rados.0.trial153.stdout:520: seq_num 251 ranges {618557=727148,2104745=113812} 2026-02-20T12:01:36.523 INFO:tasks.rados.rados.0.trial153.stdout:520: writing trial15342161-45 from 618557 to 1345705 tid 1 2026-02-20T12:01:36.527 INFO:tasks.rados.rados.0.trial153.stdout:520: writing trial15342161-45 from 2104745 to 2218557 tid 2 2026-02-20T12:01:36.527 INFO:tasks.rados.rados.0.trial153.stdout:521: snap_create 2026-02-20T12:01:36.539 INFO:tasks.rados.rados.0.trial153.stdout:518: expect (ObjNum 201 snap 50 seq_num 201) 2026-02-20T12:01:36.613 INFO:tasks.rados.rados.0.trial153.stdout:520: finishing write tid 1 to trial15342161-45 2026-02-20T12:01:36.613 INFO:tasks.rados.rados.0.trial153.stdout:520: finishing write tid 2 to trial15342161-45 2026-02-20T12:01:36.613 INFO:tasks.rados.rados.0.trial153.stdout:520: finishing write tid 3 to trial15342161-45 2026-02-20T12:01:36.613 INFO:tasks.rados.rados.0.trial153.stdout:520: finishing write tid 4 to trial15342161-45 2026-02-20T12:01:36.613 INFO:tasks.rados.rados.0.trial153.stdout:520: oid 45 updating version 0 to 51 2026-02-20T12:01:36.613 INFO:tasks.rados.rados.0.trial153.stdout:520: oid 45 version 51 is already newer than 51 2026-02-20T12:01:36.613 INFO:tasks.rados.rados.0.trial153.stdout:520: oid 45 version 51 is already newer than 51 2026-02-20T12:01:36.613 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 51 (ObjNum 251 snap 65 seq_num 251) dirty exists 2026-02-20T12:01:36.613 INFO:tasks.rados.rados.0.trial153.stdout:520: left oid 45 (ObjNum 251 snap 65 seq_num 251) 2026-02-20T12:01:36.613 INFO:tasks.rados.rados.0.trial153.stdout:517: expect (ObjNum 236 snap 60 seq_num 236) 2026-02-20T12:01:36.617 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 654769 2026-02-20T12:01:36.736 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:36.799 INFO:tasks.rados.rados.0.trial153.stdout:519: finishing write tid 1 to trial15342161-14 2026-02-20T12:01:36.799 INFO:tasks.rados.rados.0.trial153.stdout:519: finishing write tid 2 to trial15342161-14 2026-02-20T12:01:36.799 INFO:tasks.rados.rados.0.trial153.stdout:519: finishing write tid 3 to trial15342161-14 2026-02-20T12:01:36.799 INFO:tasks.rados.rados.0.trial153.stdout:519: finishing write tid 4 to trial15342161-14 2026-02-20T12:01:36.800 INFO:tasks.rados.rados.0.trial153.stdout:519: oid 14 updating version 0 to 133 2026-02-20T12:01:36.800 INFO:tasks.rados.rados.0.trial153.stdout:519: oid 14 version 133 is already newer than 133 2026-02-20T12:01:36.800 INFO:tasks.rados.rados.0.trial153.stdout:519: oid 14 version 133 is already newer than 133 2026-02-20T12:01:36.800 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 133 (ObjNum 250 snap 65 seq_num 250) dirty exists 2026-02-20T12:01:36.800 INFO:tasks.rados.rados.0.trial153.stdout:519: left oid 14 (ObjNum 250 snap 65 seq_num 250) 2026-02-20T12:01:36.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~143s 2026-02-20T12:01:37.118 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:37.195 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:37.189+0000 7f0d2805a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:37.197 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:37.192+0000 7f0d2805a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:37.224 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:37.241 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:37.511 INFO:tasks.rados.rados.0.trial153.stdout:517: done (4 left) 2026-02-20T12:01:37.511 INFO:tasks.rados.rados.0.trial153.stdout:518: done (3 left) 2026-02-20T12:01:37.511 INFO:tasks.rados.rados.0.trial153.stdout:519: done (2 left) 2026-02-20T12:01:37.512 INFO:tasks.rados.rados.0.trial153.stdout:520: done (1 left) 2026-02-20T12:01:37.512 INFO:tasks.rados.rados.0.trial153.stdout:521: done (0 left) 2026-02-20T12:01:37.512 INFO:tasks.rados.rados.0.trial153.stdout:522: read oid 10 snap -1 2026-02-20T12:01:37.512 INFO:tasks.rados.rados.0.trial153.stdout:522: expect (ObjNum 218 snap 56 seq_num 218) 2026-02-20T12:01:37.512 INFO:tasks.rados.rados.0.trial153.stdout:523: write (excl) oid 19 current snap is 66 2026-02-20T12:01:37.512 INFO:tasks.rados.rados.0.trial153.stdout:523: seq_num 252 ranges {635364=402397,1763227=472137} 2026-02-20T12:01:37.514 INFO:tasks.rados.rados.0.trial153.stdout:523: writing trial15342161-19 from 635364 to 1037761 tid 1 2026-02-20T12:01:37.519 INFO:tasks.rados.rados.0.trial153.stdout:523: writing trial15342161-19 from 1763227 to 2235364 tid 2 2026-02-20T12:01:37.520 INFO:tasks.rados.rados.0.trial153.stdout:524: read oid 49 snap -1 2026-02-20T12:01:37.520 INFO:tasks.rados.rados.0.trial153.stdout:524: expect (ObjNum 196 snap 47 seq_num 196) 2026-02-20T12:01:37.520 INFO:tasks.rados.rados.0.trial153.stdout:525: read oid 6 snap -1 2026-02-20T12:01:37.520 INFO:tasks.rados.rados.0.trial153.stdout:525: expect deleted 2026-02-20T12:01:37.520 INFO:tasks.rados.rados.0.trial153.stdout:526: snap_remove snap 65 2026-02-20T12:01:38.512 INFO:tasks.rados.rados.0.trial153.stdout:526: done (4 left) 2026-02-20T12:01:38.512 INFO:tasks.rados.rados.0.trial153.stdout:527: read oid 44 snap -1 2026-02-20T12:01:38.512 INFO:tasks.rados.rados.0.trial153.stdout:527: expect (ObjNum 226 snap 58 seq_num 226) 2026-02-20T12:01:38.512 INFO:tasks.rados.rados.0.trial153.stdout:528: read oid 43 snap 63 2026-02-20T12:01:38.512 INFO:tasks.rados.rados.0.trial153.stdout:528: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:01:38.512 INFO:tasks.rados.rados.0.trial153.stdout:529: delete oid 34 current snap is 66 2026-02-20T12:01:38.512 INFO:tasks.rados.rados.0.trial153.stdout:524: expect (ObjNum 196 snap 47 seq_num 196) 2026-02-20T12:01:38.512 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:38.550 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 142036 2026-02-20T12:01:38.625 INFO:tasks.rados.rados.0.trial153.stdout:523: finishing write tid 1 to trial15342161-19 2026-02-20T12:01:38.625 INFO:tasks.rados.rados.0.trial153.stdout:522: expect (ObjNum 218 snap 56 seq_num 218) 2026-02-20T12:01:38.681 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:38.737 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 183974 2026-02-20T12:01:38.793 INFO:tasks.rados.rados.0.trial153.stdout:523: finishing write tid 2 to trial15342161-19 2026-02-20T12:01:38.793 INFO:tasks.rados.rados.0.trial153.stdout:522: done (6 left) 2026-02-20T12:01:38.793 INFO:tasks.rados.rados.0.trial153.stdout:524: done (5 left) 2026-02-20T12:01:38.794 INFO:tasks.rados.rados.0.trial153.stdout:525: done (4 left) 2026-02-20T12:01:38.794 INFO:tasks.rados.rados.0.trial153.stdout:529: done (3 left) 2026-02-20T12:01:38.794 INFO:tasks.rados.rados.0.trial153.stdout:530: write oid 4 current snap is 66 2026-02-20T12:01:38.794 INFO:tasks.rados.rados.0.trial153.stdout:523: finishing write tid 3 to trial15342161-19 2026-02-20T12:01:38.794 INFO:tasks.rados.rados.0.trial153.stdout:530: seq_num 254 ranges {668978=552895,1763838=505140} 2026-02-20T12:01:38.797 INFO:tasks.rados.rados.0.trial153.stdout:530: writing trial15342161-4 from 668978 to 1221873 tid 1 2026-02-20T12:01:38.801 INFO:tasks.rados.rados.0.trial153.stdout:530: writing trial15342161-4 from 1763838 to 2268978 tid 2 2026-02-20T12:01:38.802 INFO:tasks.rados.rados.0.trial153.stdout:531: snap_remove snap 64 2026-02-20T12:01:38.802 INFO:tasks.rados.rados.0.trial153.stdout:523: finishing write tid 4 to trial15342161-19 2026-02-20T12:01:38.802 INFO:tasks.rados.rados.0.trial153.stdout:523: oid 19 updating version 0 to 135 2026-02-20T12:01:38.802 INFO:tasks.rados.rados.0.trial153.stdout:523: oid 19 updating version 135 to 137 2026-02-20T12:01:38.802 INFO:tasks.rados.rados.0.trial153.stdout:523: oid 19 version 137 is already newer than 137 2026-02-20T12:01:38.803 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 137 (ObjNum 252 snap 66 seq_num 252) dirty exists 2026-02-20T12:01:38.803 INFO:tasks.rados.rados.0.trial153.stdout:523: left oid 19 (ObjNum 252 snap 66 seq_num 252) 2026-02-20T12:01:39.242 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:39.320 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:39.314+0000 7ff477be4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:39.322 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:39.317+0000 7ff477be4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:39.348 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:39.364 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:39.514 INFO:tasks.rados.rados.0.trial153.stdout:523: done (4 left) 2026-02-20T12:01:39.514 INFO:tasks.rados.rados.0.trial153.stdout:531: done (3 left) 2026-02-20T12:01:39.515 INFO:tasks.rados.rados.0.trial153.stdout:532: delete oid 5 current snap is 66 2026-02-20T12:01:39.515 INFO:tasks.rados.rados.0.trial153.stdout:528: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:01:39.520 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:39.563 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1227121 2026-02-20T12:01:39.590 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:39.594 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:39.640 INFO:tasks.rados.rados.0.trial153.stdout:527: expect (ObjNum 226 snap 58 seq_num 226) 2026-02-20T12:01:39.640 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:39.725 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:39.771 INFO:tasks.rados.rados.0.trial153.stdout:527: done (3 left) 2026-02-20T12:01:39.771 INFO:tasks.rados.rados.0.trial153.stdout:528: done (2 left) 2026-02-20T12:01:39.771 INFO:tasks.rados.rados.0.trial153.stdout:532: done (1 left) 2026-02-20T12:01:39.771 INFO:tasks.rados.rados.0.trial153.stdout:533: read oid 47 snap 63 2026-02-20T12:01:39.771 INFO:tasks.rados.rados.0.trial153.stdout:533: expect (ObjNum 195 snap 47 seq_num 195) 2026-02-20T12:01:39.771 INFO:tasks.rados.rados.0.trial153.stdout:534: write oid 33 current snap is 66 2026-02-20T12:01:39.771 INFO:tasks.rados.rados.0.trial153.stdout:530: finishing write tid 1 to trial15342161-4 2026-02-20T12:01:39.771 INFO:tasks.rados.rados.0.trial153.stdout:534: seq_num 256 ranges {702592=703393,1880802=421790} 2026-02-20T12:01:39.776 INFO:tasks.rados.rados.0.trial153.stdout:534: writing trial15342161-33 from 702592 to 1405985 tid 1 2026-02-20T12:01:39.780 INFO:tasks.rados.rados.0.trial153.stdout:534: writing trial15342161-33 from 1880802 to 2302592 tid 2 2026-02-20T12:01:39.780 INFO:tasks.rados.rados.0.trial153.stdout:535: delete oid 39 current snap is 66 2026-02-20T12:01:39.780 INFO:tasks.rados.rados.0.trial153.stdout:530: finishing write tid 2 to trial15342161-4 2026-02-20T12:01:39.780 INFO:tasks.rados.rados.0.trial153.stdout:530: finishing write tid 3 to trial15342161-4 2026-02-20T12:01:39.780 INFO:tasks.rados.rados.0.trial153.stdout:530: finishing write tid 4 to trial15342161-4 2026-02-20T12:01:39.780 INFO:tasks.rados.rados.0.trial153.stdout:530: oid 4 updating version 0 to 87 2026-02-20T12:01:39.780 INFO:tasks.rados.rados.0.trial153.stdout:530: oid 4 version 87 is already newer than 87 2026-02-20T12:01:39.780 INFO:tasks.rados.rados.0.trial153.stdout:530: oid 4 version 87 is already newer than 87 2026-02-20T12:01:39.780 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 4 v 87 (ObjNum 254 snap 66 seq_num 254) dirty exists 2026-02-20T12:01:39.780 INFO:tasks.rados.rados.0.trial153.stdout:530: left oid 4 (ObjNum 254 snap 66 seq_num 254) 2026-02-20T12:01:39.790 INFO:tasks.rados.rados.0.trial153.stdout:533: expect (ObjNum 195 snap 47 seq_num 195) 2026-02-20T12:01:39.790 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:39.819 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 192026 2026-02-20T12:01:39.873 INFO:tasks.rados.rados.0.trial153.stdout:530: done (3 left) 2026-02-20T12:01:39.874 INFO:tasks.rados.rados.0.trial153.stdout:533: done (2 left) 2026-02-20T12:01:39.874 INFO:tasks.rados.rados.0.trial153.stdout:535: done (1 left) 2026-02-20T12:01:39.874 INFO:tasks.rados.rados.0.trial153.stdout:536: read oid 38 snap -1 2026-02-20T12:01:39.874 INFO:tasks.rados.rados.0.trial153.stdout:536: expect (ObjNum 166 snap 36 seq_num 166) 2026-02-20T12:01:39.874 INFO:tasks.rados.rados.0.trial153.stdout:537: write oid 48 current snap is 66 2026-02-20T12:01:39.874 INFO:tasks.rados.rados.0.trial153.stdout:534: finishing write tid 1 to trial15342161-33 2026-02-20T12:01:39.874 INFO:tasks.rados.rados.0.trial153.stdout:534: finishing write tid 2 to trial15342161-33 2026-02-20T12:01:39.874 INFO:tasks.rados.rados.0.trial153.stdout:534: finishing write tid 3 to trial15342161-33 2026-02-20T12:01:39.874 INFO:tasks.rados.rados.0.trial153.stdout:534: finishing write tid 4 to trial15342161-33 2026-02-20T12:01:39.875 INFO:tasks.rados.rados.0.trial153.stdout:534: oid 33 updating version 0 to 61 2026-02-20T12:01:39.875 INFO:tasks.rados.rados.0.trial153.stdout:534: oid 33 version 61 is already newer than 61 2026-02-20T12:01:39.875 INFO:tasks.rados.rados.0.trial153.stdout:534: oid 33 version 61 is already newer than 61 2026-02-20T12:01:39.875 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 33 v 61 (ObjNum 256 snap 66 seq_num 256) dirty exists 2026-02-20T12:01:39.875 INFO:tasks.rados.rados.0.trial153.stdout:534: left oid 33 (ObjNum 256 snap 66 seq_num 256) 2026-02-20T12:01:39.875 INFO:tasks.rados.rados.0.trial153.stdout:537: seq_num 258 ranges {736206=453891,1881413=454793} 2026-02-20T12:01:39.877 INFO:tasks.rados.rados.0.trial153.stdout:537: writing trial15342161-48 from 736206 to 1190097 tid 1 2026-02-20T12:01:39.882 INFO:tasks.rados.rados.0.trial153.stdout:537: writing trial15342161-48 from 1881413 to 2336206 tid 2 2026-02-20T12:01:39.882 INFO:tasks.rados.rados.0.trial153.stdout:534: done (2 left) 2026-02-20T12:01:39.882 INFO:tasks.rados.rados.0.trial153.stdout:538: snap_create 2026-02-20T12:01:39.897 INFO:tasks.rados.rados.0.trial153.stdout:536: expect (ObjNum 166 snap 36 seq_num 166) 2026-02-20T12:01:39.951 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2275365 2026-02-20T12:01:39.961 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:39.999 INFO:tasks.rados.rados.0.trial153.stdout:537: finishing write tid 1 to trial15342161-48 2026-02-20T12:01:40.000 INFO:tasks.rados.rados.0.trial153.stdout:537: finishing write tid 2 to trial15342161-48 2026-02-20T12:01:40.000 INFO:tasks.rados.rados.0.trial153.stdout:537: finishing write tid 3 to trial15342161-48 2026-02-20T12:01:40.000 INFO:tasks.rados.rados.0.trial153.stdout:537: finishing write tid 4 to trial15342161-48 2026-02-20T12:01:40.000 INFO:tasks.rados.rados.0.trial153.stdout:537: oid 48 updating version 0 to 38 2026-02-20T12:01:40.000 INFO:tasks.rados.rados.0.trial153.stdout:537: oid 48 version 38 is already newer than 38 2026-02-20T12:01:40.000 INFO:tasks.rados.rados.0.trial153.stdout:537: oid 48 version 38 is already newer than 38 2026-02-20T12:01:40.000 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 48 v 38 (ObjNum 258 snap 66 seq_num 258) dirty exists 2026-02-20T12:01:40.001 INFO:tasks.rados.rados.0.trial153.stdout:537: left oid 48 (ObjNum 258 snap 66 seq_num 258) 2026-02-20T12:01:40.517 INFO:tasks.rados.rados.0.trial153.stdout:536: done (2 left) 2026-02-20T12:01:40.517 INFO:tasks.rados.rados.0.trial153.stdout:537: done (1 left) 2026-02-20T12:01:40.517 INFO:tasks.rados.rados.0.trial153.stdout:538: done (0 left) 2026-02-20T12:01:40.518 INFO:tasks.rados.rados.0.trial153.stdout:539: read oid 15 snap 66 2026-02-20T12:01:40.518 INFO:tasks.rados.rados.0.trial153.stdout:539: expect (ObjNum 161 snap 34 seq_num 161) 2026-02-20T12:01:40.518 INFO:tasks.rados.rados.0.trial153.stdout:540: read oid 50 snap -1 2026-02-20T12:01:40.518 INFO:tasks.rados.rados.0.trial153.stdout:540: expect (ObjNum 234 snap 60 seq_num 234) 2026-02-20T12:01:40.518 INFO:tasks.rados.rados.0.trial153.stdout:541: snap_create 2026-02-20T12:01:40.579 INFO:tasks.rados.rados.0.trial153.stdout:539: expect (ObjNum 161 snap 34 seq_num 161) 2026-02-20T12:01:40.664 INFO:tasks.rados.rados.0.trial153.stdout:540: expect (ObjNum 234 snap 60 seq_num 234) 2026-02-20T12:01:40.797 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:41.365 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:41.443 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:41.437+0000 7fe77a7ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:41.445 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:41.441+0000 7fe77a7ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:41.471 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:41.488 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:41.489 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:41.526 INFO:tasks.rados.rados.0.trial153.stdout:539: done (2 left) 2026-02-20T12:01:41.526 INFO:tasks.rados.rados.0.trial153.stdout:540: done (1 left) 2026-02-20T12:01:41.527 INFO:tasks.rados.rados.0.trial153.stdout:541: done (0 left) 2026-02-20T12:01:41.527 INFO:tasks.rados.rados.0.trial153.stdout:542: write (excl) oid 19 current snap is 68 2026-02-20T12:01:41.527 INFO:tasks.rados.rados.0.trial153.stdout:542: seq_num 259 ranges {753013=645493,2056248=296765} 2026-02-20T12:01:41.532 INFO:tasks.rados.rados.0.trial153.stdout:542: writing trial15342161-19 from 753013 to 1398506 tid 1 2026-02-20T12:01:41.536 INFO:tasks.rados.rados.0.trial153.stdout:542: writing trial15342161-19 from 2056248 to 2353013 tid 2 2026-02-20T12:01:41.536 INFO:tasks.rados.rados.0.trial153.stdout:543: write oid 9 current snap is 68 2026-02-20T12:01:41.536 INFO:tasks.rados.rados.0.trial153.stdout:543: seq_num 260 ranges {769820=720742,2114730=255090} 2026-02-20T12:01:41.542 INFO:tasks.rados.rados.0.trial153.stdout:543: writing trial15342161-9 from 769820 to 1490562 tid 1 2026-02-20T12:01:41.545 INFO:tasks.rados.rados.0.trial153.stdout:543: writing trial15342161-9 from 2114730 to 2369820 tid 2 2026-02-20T12:01:41.546 INFO:tasks.rados.rados.0.trial153.stdout:544: delete oid 5 current snap is 68 2026-02-20T12:01:41.548 INFO:tasks.rados.rados.0.trial153.stdout:544: done (2 left) 2026-02-20T12:01:41.548 INFO:tasks.rados.rados.0.trial153.stdout:545: snap_remove snap 66 2026-02-20T12:01:42.419 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~149s 2026-02-20T12:01:42.524 INFO:tasks.rados.rados.0.trial153.stdout:545: done (2 left) 2026-02-20T12:01:42.524 INFO:tasks.rados.rados.0.trial153.stdout:546: read oid 6 snap -1 2026-02-20T12:01:42.524 INFO:tasks.rados.rados.0.trial153.stdout:546: expect deleted 2026-02-20T12:01:42.524 INFO:tasks.rados.rados.0.trial153.stdout:547: snap_create 2026-02-20T12:01:42.524 INFO:tasks.rados.rados.0.trial153.stdout:542: finishing write tid 1 to trial15342161-19 2026-02-20T12:01:42.524 INFO:tasks.rados.rados.0.trial153.stdout:542: finishing write tid 2 to trial15342161-19 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:543: finishing write tid 1 to trial15342161-9 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:542: finishing write tid 3 to trial15342161-19 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:542: finishing write tid 4 to trial15342161-19 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:542: oid 19 updating version 0 to 147 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:542: oid 19 version 147 is already newer than 147 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:542: oid 19 version 147 is already newer than 145 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 147 (ObjNum 259 snap 68 seq_num 259) dirty exists 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:542: left oid 19 (ObjNum 259 snap 68 seq_num 259) 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:543: finishing write tid 2 to trial15342161-9 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:543: finishing write tid 3 to trial15342161-9 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:543: finishing write tid 4 to trial15342161-9 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:543: oid 9 updating version 0 to 203 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:543: oid 9 version 203 is already newer than 203 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:543: oid 9 version 203 is already newer than 203 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 9 v 203 (ObjNum 260 snap 68 seq_num 260) dirty exists 2026-02-20T12:01:42.525 INFO:tasks.rados.rados.0.trial153.stdout:543: left oid 9 (ObjNum 260 snap 68 seq_num 260) 2026-02-20T12:01:43.490 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:43.527 INFO:tasks.rados.rados.0.trial153.stdout:542: done (3 left) 2026-02-20T12:01:43.527 INFO:tasks.rados.rados.0.trial153.stdout:543: done (2 left) 2026-02-20T12:01:43.528 INFO:tasks.rados.rados.0.trial153.stdout:546: done (1 left) 2026-02-20T12:01:43.528 INFO:tasks.rados.rados.0.trial153.stdout:547: done (0 left) 2026-02-20T12:01:43.528 INFO:tasks.rados.rados.0.trial153.stdout:548: write (excl) oid 25 current snap is 69 2026-02-20T12:01:43.528 INFO:tasks.rados.rados.0.trial153.stdout:548: seq_num 262 ranges {403434=471240,1315341=400756} 2026-02-20T12:01:43.530 INFO:tasks.rados.rados.0.trial153.stdout:548: writing trial15342161-25 from 403434 to 874674 tid 1 2026-02-20T12:01:43.533 INFO:tasks.rados.rados.0.trial153.stdout:548: writing trial15342161-25 from 1315341 to 1716097 tid 2 2026-02-20T12:01:43.534 INFO:tasks.rados.rados.0.trial153.stdout:549: write oid 45 current snap is 69 2026-02-20T12:01:43.534 INFO:tasks.rados.rados.0.trial153.stdout:549: seq_num 263 ranges {420241=546489,1373823=544414} 2026-02-20T12:01:43.537 INFO:tasks.rados.rados.0.trial153.stdout:549: writing trial15342161-45 from 420241 to 966730 tid 1 2026-02-20T12:01:43.541 INFO:tasks.rados.rados.0.trial153.stdout:549: writing trial15342161-45 from 1373823 to 1918237 tid 2 2026-02-20T12:01:43.541 INFO:tasks.rados.rados.0.trial153.stdout:550: snap_remove snap 67 2026-02-20T12:01:43.571 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:43.566+0000 7fa928da4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:43.574 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:43.570+0000 7fa928da4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:43.608 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:43.624 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:44.534 INFO:tasks.rados.rados.0.trial153.stdout:550: done (2 left) 2026-02-20T12:01:44.534 INFO:tasks.rados.rados.0.trial153.stdout:551: delete oid 5 current snap is 69 2026-02-20T12:01:44.534 INFO:tasks.rados.rados.0.trial153.stdout:548: finishing write tid 1 to trial15342161-25 2026-02-20T12:01:44.534 INFO:tasks.rados.rados.0.trial153.stdout:548: finishing write tid 2 to trial15342161-25 2026-02-20T12:01:44.535 INFO:tasks.rados.rados.0.trial153.stdout:548: finishing write tid 3 to trial15342161-25 2026-02-20T12:01:44.535 INFO:tasks.rados.rados.0.trial153.stdout:549: finishing write tid 1 to trial15342161-45 2026-02-20T12:01:44.535 INFO:tasks.rados.rados.0.trial153.stdout:548: finishing write tid 4 to trial15342161-25 2026-02-20T12:01:44.535 INFO:tasks.rados.rados.0.trial153.stdout:548: oid 25 updating version 0 to 38 2026-02-20T12:01:44.535 INFO:tasks.rados.rados.0.trial153.stdout:548: oid 25 version 38 is already newer than 36 2026-02-20T12:01:44.535 INFO:tasks.rados.rados.0.trial153.stdout:548: oid 25 version 38 is already newer than 38 2026-02-20T12:01:44.535 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 25 v 38 (ObjNum 262 snap 69 seq_num 262) dirty exists 2026-02-20T12:01:44.535 INFO:tasks.rados.rados.0.trial153.stdout:548: left oid 25 (ObjNum 262 snap 69 seq_num 262) 2026-02-20T12:01:44.535 INFO:tasks.rados.rados.0.trial153.stdout:549: finishing write tid 2 to trial15342161-45 2026-02-20T12:01:44.535 INFO:tasks.rados.rados.0.trial153.stdout:549: finishing write tid 3 to trial15342161-45 2026-02-20T12:01:44.536 INFO:tasks.rados.rados.0.trial153.stdout:549: finishing write tid 4 to trial15342161-45 2026-02-20T12:01:44.536 INFO:tasks.rados.rados.0.trial153.stdout:549: oid 45 updating version 0 to 55 2026-02-20T12:01:44.536 INFO:tasks.rados.rados.0.trial153.stdout:549: oid 45 version 55 is already newer than 55 2026-02-20T12:01:44.536 INFO:tasks.rados.rados.0.trial153.stdout:549: oid 45 version 55 is already newer than 55 2026-02-20T12:01:44.536 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 55 (ObjNum 263 snap 69 seq_num 263) dirty exists 2026-02-20T12:01:44.536 INFO:tasks.rados.rados.0.trial153.stdout:549: left oid 45 (ObjNum 263 snap 69 seq_num 263) 2026-02-20T12:01:44.560 INFO:tasks.rados.rados.0.trial153.stdout:548: done (2 left) 2026-02-20T12:01:44.560 INFO:tasks.rados.rados.0.trial153.stdout:549: done (1 left) 2026-02-20T12:01:44.560 INFO:tasks.rados.rados.0.trial153.stdout:551: done (0 left) 2026-02-20T12:01:44.561 INFO:tasks.rados.rados.0.trial153.stdout:552: write (excl) oid 50 current snap is 69 2026-02-20T12:01:44.561 INFO:tasks.rados.rados.0.trial153.stdout:552: seq_num 265 ranges {453855=696987,1890787=548083} 2026-02-20T12:01:44.565 INFO:tasks.rados.rados.0.trial153.stdout:552: writing trial15342161-50 from 453855 to 1150842 tid 1 2026-02-20T12:01:44.570 INFO:tasks.rados.rados.0.trial153.stdout:552: writing trial15342161-50 from 1890787 to 2438870 tid 2 2026-02-20T12:01:44.570 INFO:tasks.rados.rados.0.trial153.stdout:553: snap_create 2026-02-20T12:01:44.595 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:44.598 INFO:tasks.rados.rados.0.trial153.stdout:552: finishing write tid 1 to trial15342161-50 2026-02-20T12:01:44.621 INFO:tasks.rados.rados.0.trial153.stdout:552: finishing write tid 2 to trial15342161-50 2026-02-20T12:01:44.630 INFO:tasks.rados.rados.0.trial153.stdout:552: finishing write tid 3 to trial15342161-50 2026-02-20T12:01:44.632 INFO:tasks.rados.rados.0.trial153.stdout:552: finishing write tid 4 to trial15342161-50 2026-02-20T12:01:44.632 INFO:tasks.rados.rados.0.trial153.stdout:552: oid 50 updating version 0 to 151 2026-02-20T12:01:44.633 INFO:tasks.rados.rados.0.trial153.stdout:552: oid 50 updating version 151 to 153 2026-02-20T12:01:44.633 INFO:tasks.rados.rados.0.trial153.stdout:552: oid 50 version 153 is already newer than 153 2026-02-20T12:01:44.633 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 50 v 153 (ObjNum 265 snap 69 seq_num 265) dirty exists 2026-02-20T12:01:44.633 INFO:tasks.rados.rados.0.trial153.stdout:552: left oid 50 (ObjNum 265 snap 69 seq_num 265) 2026-02-20T12:01:44.666 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:44.669 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:45.538 INFO:tasks.rados.rados.0.trial153.stdout:552: done (1 left) 2026-02-20T12:01:45.539 INFO:tasks.rados.rados.0.trial153.stdout:553: done (0 left) 2026-02-20T12:01:45.539 INFO:tasks.rados.rados.0.trial153.stdout:554: read oid 17 snap -1 2026-02-20T12:01:45.539 INFO:tasks.rados.rados.0.trial153.stdout:554: expect (ObjNum 191 snap 47 seq_num 191) 2026-02-20T12:01:45.539 INFO:tasks.rados.rados.0.trial153.stdout:555: write oid 34 current snap is 70 2026-02-20T12:01:45.539 INFO:tasks.rados.rados.0.trial153.stdout:555: seq_num 266 ranges {470662=772236,1832916=408094} 2026-02-20T12:01:45.543 INFO:tasks.rados.rados.0.trial153.stdout:555: writing trial15342161-34 from 470662 to 1242898 tid 1 2026-02-20T12:01:45.547 INFO:tasks.rados.rados.0.trial153.stdout:555: writing trial15342161-34 from 1832916 to 2241010 tid 2 2026-02-20T12:01:45.548 INFO:tasks.rados.rados.0.trial153.stdout:556: write oid 33 current snap is 70 2026-02-20T12:01:45.548 INFO:tasks.rados.rados.0.trial153.stdout:556: seq_num 267 ranges {487469=563838,1607751=551752} 2026-02-20T12:01:45.551 INFO:tasks.rados.rados.0.trial153.stdout:556: writing trial15342161-33 from 487469 to 1051307 tid 1 2026-02-20T12:01:45.556 INFO:tasks.rados.rados.0.trial153.stdout:556: writing trial15342161-33 from 1607751 to 2159503 tid 2 2026-02-20T12:01:45.556 INFO:tasks.rados.rados.0.trial153.stdout:557: read oid 28 snap -1 2026-02-20T12:01:45.556 INFO:tasks.rados.rados.0.trial153.stdout:557: expect (ObjNum 203 snap 51 seq_num 203) 2026-02-20T12:01:45.556 INFO:tasks.rados.rados.0.trial153.stdout:558: write (excl) oid 31 current snap is 70 2026-02-20T12:01:45.556 INFO:tasks.rados.rados.0.trial153.stdout:558: seq_num 268 ranges {504276=639087,1666233=695410} 2026-02-20T12:01:45.560 INFO:tasks.rados.rados.0.trial153.stdout:558: writing trial15342161-31 from 504276 to 1143363 tid 1 2026-02-20T12:01:45.565 INFO:tasks.rados.rados.0.trial153.stdout:558: writing trial15342161-31 from 1666233 to 2361643 tid 2 2026-02-20T12:01:45.565 INFO:tasks.rados.rados.0.trial153.stdout:559: delete oid 48 current snap is 70 2026-02-20T12:01:45.567 INFO:tasks.rados.rados.0.trial153.stdout:554: expect (ObjNum 191 snap 47 seq_num 191) 2026-02-20T12:01:45.567 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:45.625 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:45.667 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2145604 2026-02-20T12:01:45.688 INFO:tasks.rados.rados.0.trial153.stdout:554: done (5 left) 2026-02-20T12:01:45.688 INFO:tasks.rados.rados.0.trial153.stdout:559: done (4 left) 2026-02-20T12:01:45.689 INFO:tasks.rados.rados.0.trial153.stdout:560: write oid 46 current snap is 70 2026-02-20T12:01:45.689 INFO:tasks.rados.rados.0.trial153.stdout:557: expect (ObjNum 203 snap 51 seq_num 203) 2026-02-20T12:01:45.689 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:45.703 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:45.697+0000 7ff2fb75b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:45.706 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:45.701+0000 7ff2fb75b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:45.726 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:45.734 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:45.749 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:45.796 INFO:tasks.rados.rados.0.trial153.stdout:555: finishing write tid 1 to trial15342161-34 2026-02-20T12:01:45.796 INFO:tasks.rados.rados.0.trial153.stdout:556: finishing write tid 1 to trial15342161-33 2026-02-20T12:01:45.797 INFO:tasks.rados.rados.0.trial153.stdout:555: finishing write tid 2 to trial15342161-34 2026-02-20T12:01:45.797 INFO:tasks.rados.rados.0.trial153.stdout:556: finishing write tid 2 to trial15342161-33 2026-02-20T12:01:45.797 INFO:tasks.rados.rados.0.trial153.stdout:556: finishing write tid 3 to trial15342161-33 2026-02-20T12:01:45.797 INFO:tasks.rados.rados.0.trial153.stdout:555: finishing write tid 3 to trial15342161-34 2026-02-20T12:01:45.797 INFO:tasks.rados.rados.0.trial153.stdout:560: seq_num 270 ranges {537890=789585,1783197=699079} 2026-02-20T12:01:45.801 INFO:tasks.rados.rados.0.trial153.stdout:560: writing trial15342161-46 from 537890 to 1327475 tid 1 2026-02-20T12:01:45.805 INFO:tasks.rados.rados.0.trial153.stdout:560: writing trial15342161-46 from 1783197 to 2482276 tid 2 2026-02-20T12:01:45.805 INFO:tasks.rados.rados.0.trial153.stdout:557: done (4 left) 2026-02-20T12:01:45.806 INFO:tasks.rados.rados.0.trial153.stdout:561: read oid 10 snap 63 2026-02-20T12:01:45.806 INFO:tasks.rados.rados.0.trial153.stdout:561: expect (ObjNum 218 snap 56 seq_num 218) 2026-02-20T12:01:45.806 INFO:tasks.rados.rados.0.trial153.stdout:562: snap_remove snap 68 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:562: done (5 left) 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:563: delete oid 16 current snap is 70 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:555: finishing write tid 4 to trial15342161-34 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:555: oid 34 updating version 0 to 96 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:555: oid 34 version 96 is already newer than 96 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:555: oid 34 version 96 is already newer than 96 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 96 (ObjNum 266 snap 70 seq_num 266) dirty exists 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:555: left oid 34 (ObjNum 266 snap 70 seq_num 266) 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:556: finishing write tid 4 to trial15342161-33 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:556: oid 33 updating version 0 to 65 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:556: oid 33 version 65 is already newer than 65 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:556: oid 33 version 65 is already newer than 65 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 33 v 65 (ObjNum 267 snap 70 seq_num 267) dirty exists 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:556: left oid 33 (ObjNum 267 snap 70 seq_num 267) 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:558: finishing write tid 1 to trial15342161-31 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:558: finishing write tid 2 to trial15342161-31 2026-02-20T12:01:46.511 INFO:tasks.rados.rados.0.trial153.stdout:558: finishing write tid 3 to trial15342161-31 2026-02-20T12:01:46.512 INFO:tasks.rados.rados.0.trial153.stdout:558: finishing write tid 4 to trial15342161-31 2026-02-20T12:01:46.512 INFO:tasks.rados.rados.0.trial153.stdout:558: oid 31 updating version 0 to 42 2026-02-20T12:01:46.512 INFO:tasks.rados.rados.0.trial153.stdout:558: oid 31 version 42 is already newer than 40 2026-02-20T12:01:46.512 INFO:tasks.rados.rados.0.trial153.stdout:558: oid 31 version 42 is already newer than 42 2026-02-20T12:01:46.512 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 31 v 42 (ObjNum 268 snap 70 seq_num 268) dirty exists 2026-02-20T12:01:46.512 INFO:tasks.rados.rados.0.trial153.stdout:558: left oid 31 (ObjNum 268 snap 70 seq_num 268) 2026-02-20T12:01:46.512 INFO:tasks.rados.rados.0.trial153.stdout:561: expect (ObjNum 218 snap 56 seq_num 218) 2026-02-20T12:01:46.513 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 513571 2026-02-20T12:01:46.664 INFO:tasks.rados.rados.0.trial153.stdout:560: finishing write tid 1 to trial15342161-46 2026-02-20T12:01:46.664 INFO:tasks.rados.rados.0.trial153.stdout:560: finishing write tid 2 to trial15342161-46 2026-02-20T12:01:46.664 INFO:tasks.rados.rados.0.trial153.stdout:560: finishing write tid 3 to trial15342161-46 2026-02-20T12:01:46.664 INFO:tasks.rados.rados.0.trial153.stdout:560: finishing write tid 4 to trial15342161-46 2026-02-20T12:01:46.664 INFO:tasks.rados.rados.0.trial153.stdout:560: oid 46 updating version 0 to 22 2026-02-20T12:01:46.664 INFO:tasks.rados.rados.0.trial153.stdout:560: oid 46 version 22 is already newer than 22 2026-02-20T12:01:46.664 INFO:tasks.rados.rados.0.trial153.stdout:560: oid 46 version 22 is already newer than 22 2026-02-20T12:01:46.664 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 46 v 22 (ObjNum 270 snap 70 seq_num 270) dirty exists 2026-02-20T12:01:46.665 INFO:tasks.rados.rados.0.trial153.stdout:560: left oid 46 (ObjNum 270 snap 70 seq_num 270) 2026-02-20T12:01:46.665 INFO:tasks.rados.rados.0.trial153.stdout:555: done (5 left) 2026-02-20T12:01:46.665 INFO:tasks.rados.rados.0.trial153.stdout:556: done (4 left) 2026-02-20T12:01:46.665 INFO:tasks.rados.rados.0.trial153.stdout:558: done (3 left) 2026-02-20T12:01:46.665 INFO:tasks.rados.rados.0.trial153.stdout:560: done (2 left) 2026-02-20T12:01:46.665 INFO:tasks.rados.rados.0.trial153.stdout:561: done (1 left) 2026-02-20T12:01:46.665 INFO:tasks.rados.rados.0.trial153.stdout:563: done (0 left) 2026-02-20T12:01:46.665 INFO:tasks.rados.rados.0.trial153.stdout:564: snap_remove snap 63 2026-02-20T12:01:47.539 INFO:tasks.rados.rados.0.trial153.stdout:564: done (0 left) 2026-02-20T12:01:47.539 INFO:tasks.rados.rados.0.trial153.stdout:565: snap_remove snap 69 2026-02-20T12:01:47.750 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:47.827 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:47.821+0000 7fb54a9ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:47.830 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:47.826+0000 7fb54a9ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:47.857 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:47.873 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:47.922 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~154s 2026-02-20T12:01:48.542 INFO:tasks.rados.rados.0.trial153.stdout:565: done (0 left) 2026-02-20T12:01:48.542 INFO:tasks.rados.rados.0.trial153.stdout:566: write oid 26 current snap is 70 2026-02-20T12:01:48.542 INFO:tasks.rados.rados.0.trial153.stdout:566: seq_num 272 ranges {571504=540083,1783808=702748} 2026-02-20T12:01:48.546 INFO:tasks.rados.rados.0.trial153.stdout:566: writing trial15342161-26 from 571504 to 1111587 tid 1 2026-02-20T12:01:48.552 INFO:tasks.rados.rados.0.trial153.stdout:566: writing trial15342161-26 from 1783808 to 2486556 tid 2 2026-02-20T12:01:48.552 INFO:tasks.rados.rados.0.trial153.stdout:567: read oid 29 snap -1 2026-02-20T12:01:48.552 INFO:tasks.rados.rados.0.trial153.stdout:567: expect (ObjNum 148 snap 26 seq_num 148) 2026-02-20T12:01:48.552 INFO:tasks.rados.rados.0.trial153.stdout:568: delete oid 39 current snap is 70 2026-02-20T12:01:48.582 INFO:tasks.rados.rados.0.trial153.stdout:567: expect (ObjNum 148 snap 26 seq_num 148) 2026-02-20T12:01:48.603 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:48.633 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:48.663 INFO:tasks.rados.rados.0.trial153.stdout:567: done (2 left) 2026-02-20T12:01:48.663 INFO:tasks.rados.rados.0.trial153.stdout:568: done (1 left) 2026-02-20T12:01:48.663 INFO:tasks.rados.rados.0.trial153.stdout:569: write (excl) oid 41 current snap is 70 2026-02-20T12:01:48.663 INFO:tasks.rados.rados.0.trial153.stdout:566: finishing write tid 1 to trial15342161-26 2026-02-20T12:01:48.663 INFO:tasks.rados.rados.0.trial153.stdout:569: seq_num 274 ranges {605118=406934,1617125=706417} 2026-02-20T12:01:48.667 INFO:tasks.rados.rados.0.trial153.stdout:569: writing trial15342161-41 from 605118 to 1012052 tid 1 2026-02-20T12:01:48.672 INFO:tasks.rados.rados.0.trial153.stdout:569: writing trial15342161-41 from 1617125 to 2323542 tid 2 2026-02-20T12:01:48.672 INFO:tasks.rados.rados.0.trial153.stdout:570: write oid 40 current snap is 70 2026-02-20T12:01:48.672 INFO:tasks.rados.rados.0.trial153.stdout:570: seq_num 275 ranges {621925=482183,1559254=566428} 2026-02-20T12:01:48.676 INFO:tasks.rados.rados.0.trial153.stdout:570: writing trial15342161-40 from 621925 to 1104108 tid 1 2026-02-20T12:01:48.680 INFO:tasks.rados.rados.0.trial153.stdout:570: writing trial15342161-40 from 1559254 to 2125682 tid 2 2026-02-20T12:01:48.680 INFO:tasks.rados.rados.0.trial153.stdout:571: write (excl) oid 4 current snap is 70 2026-02-20T12:01:48.680 INFO:tasks.rados.rados.0.trial153.stdout:571: seq_num 276 ranges {638732=557432,1617736=710086} 2026-02-20T12:01:48.684 INFO:tasks.rados.rados.0.trial153.stdout:571: writing trial15342161-4 from 638732 to 1196164 tid 1 2026-02-20T12:01:48.689 INFO:tasks.rados.rados.0.trial153.stdout:571: writing trial15342161-4 from 1617736 to 2327822 tid 2 2026-02-20T12:01:48.689 INFO:tasks.rados.rados.0.trial153.stdout:572: write (excl) oid 6 current snap is 70 2026-02-20T12:01:48.689 INFO:tasks.rados.rados.0.trial153.stdout:572: seq_num 277 ranges {655539=632681,2076218=570097} 2026-02-20T12:01:48.694 INFO:tasks.rados.rados.0.trial153.stdout:572: writing trial15342161-6 from 655539 to 1288220 tid 1 2026-02-20T12:01:48.700 INFO:tasks.rados.rados.0.trial153.stdout:572: writing trial15342161-6 from 2076218 to 2646315 tid 2 2026-02-20T12:01:48.701 INFO:tasks.rados.rados.0.trial153.stdout:573: write (excl) oid 46 current snap is 70 2026-02-20T12:01:48.701 INFO:tasks.rados.rados.0.trial153.stdout:566: finishing write tid 2 to trial15342161-26 2026-02-20T12:01:48.701 INFO:tasks.rados.rados.0.trial153.stdout:573: seq_num 278 ranges {672346=707930,2134700=537646} 2026-02-20T12:01:48.706 INFO:tasks.rados.rados.0.trial153.stdout:573: writing trial15342161-46 from 672346 to 1380276 tid 1 2026-02-20T12:01:48.711 INFO:tasks.rados.rados.0.trial153.stdout:573: writing trial15342161-46 from 2134700 to 2672346 tid 2 2026-02-20T12:01:48.711 INFO:tasks.rados.rados.0.trial153.stdout:574: write (excl) oid 33 current snap is 70 2026-02-20T12:01:48.711 INFO:tasks.rados.rados.0.trial153.stdout:566: finishing write tid 3 to trial15342161-26 2026-02-20T12:01:48.711 INFO:tasks.rados.rados.0.trial153.stdout:574: seq_num 279 ranges {689153=783179,2076829=457413} 2026-02-20T12:01:48.716 INFO:tasks.rados.rados.0.trial153.stdout:574: writing trial15342161-33 from 689153 to 1472332 tid 1 2026-02-20T12:01:48.720 INFO:tasks.rados.rados.0.trial153.stdout:574: writing trial15342161-33 from 2076829 to 2534242 tid 2 2026-02-20T12:01:48.721 INFO:tasks.rados.rados.0.trial153.stdout:575: write (excl) oid 42 current snap is 70 2026-02-20T12:01:48.721 INFO:tasks.rados.rados.0.trial153.stdout:566: finishing write tid 4 to trial15342161-26 2026-02-20T12:01:48.721 INFO:tasks.rados.rados.0.trial153.stdout:566: oid 26 updating version 0 to 140 2026-02-20T12:01:48.721 INFO:tasks.rados.rados.0.trial153.stdout:566: oid 26 version 140 is already newer than 140 2026-02-20T12:01:48.721 INFO:tasks.rados.rados.0.trial153.stdout:566: oid 26 version 140 is already newer than 140 2026-02-20T12:01:48.721 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 26 v 140 (ObjNum 272 snap 70 seq_num 272) dirty exists 2026-02-20T12:01:48.721 INFO:tasks.rados.rados.0.trial153.stdout:566: left oid 26 (ObjNum 272 snap 70 seq_num 272) 2026-02-20T12:01:48.721 INFO:tasks.rados.rados.0.trial153.stdout:575: seq_num 280 ranges {705960=458428,1735311=717424} 2026-02-20T12:01:48.725 INFO:tasks.rados.rados.0.trial153.stdout:575: writing trial15342161-42 from 705960 to 1164388 tid 1 2026-02-20T12:01:48.730 INFO:tasks.rados.rados.0.trial153.stdout:575: writing trial15342161-42 from 1735311 to 2452735 tid 2 2026-02-20T12:01:48.730 INFO:tasks.rados.rados.0.trial153.stdout:566: done (7 left) 2026-02-20T12:01:48.730 INFO:tasks.rados.rados.0.trial153.stdout:576: read oid 49 snap -1 2026-02-20T12:01:48.730 INFO:tasks.rados.rados.0.trial153.stdout:576: expect (ObjNum 196 snap 47 seq_num 196) 2026-02-20T12:01:48.730 INFO:tasks.rados.rados.0.trial153.stdout:577: write oid 13 current snap is 70 2026-02-20T12:01:48.731 INFO:tasks.rados.rados.0.trial153.stdout:569: finishing write tid 1 to trial15342161-41 2026-02-20T12:01:48.731 INFO:tasks.rados.rados.0.trial153.stdout:577: seq_num 281 ranges {722767=533677,1793793=461082} 2026-02-20T12:01:48.735 INFO:tasks.rados.rados.0.trial153.stdout:577: writing trial15342161-13 from 722767 to 1256444 tid 1 2026-02-20T12:01:48.739 INFO:tasks.rados.rados.0.trial153.stdout:577: writing trial15342161-13 from 1793793 to 2254875 tid 2 2026-02-20T12:01:48.739 INFO:tasks.rados.rados.0.trial153.stdout:578: read oid 19 snap -1 2026-02-20T12:01:48.739 INFO:tasks.rados.rados.0.trial153.stdout:578: expect (ObjNum 259 snap 68 seq_num 259) 2026-02-20T12:01:48.739 INFO:tasks.rados.rados.0.trial153.stdout:579: write (excl) oid 26 current snap is 70 2026-02-20T12:01:48.739 INFO:tasks.rados.rados.0.trial153.stdout:571: finishing write tid 1 to trial15342161-4 2026-02-20T12:01:48.740 INFO:tasks.rados.rados.0.trial153.stdout:579: seq_num 282 ranges {739574=725279,1968628=721093} 2026-02-20T12:01:48.745 INFO:tasks.rados.rados.0.trial153.stdout:579: writing trial15342161-26 from 739574 to 1464853 tid 1 2026-02-20T12:01:48.750 INFO:tasks.rados.rados.0.trial153.stdout:579: writing trial15342161-26 from 1968628 to 2689721 tid 2 2026-02-20T12:01:48.750 INFO:tasks.rados.rados.0.trial153.stdout:580: write oid 12 current snap is 70 2026-02-20T12:01:48.750 INFO:tasks.rados.rados.0.trial153.stdout:569: finishing write tid 2 to trial15342161-41 2026-02-20T12:01:48.750 INFO:tasks.rados.rados.0.trial153.stdout:580: seq_num 283 ranges {756381=400528,1910757=464751} 2026-02-20T12:01:48.755 INFO:tasks.rados.rados.0.trial153.stdout:580: writing trial15342161-12 from 756381 to 1156909 tid 1 2026-02-20T12:01:48.761 INFO:tasks.rados.rados.0.trial153.stdout:580: writing trial15342161-12 from 1910757 to 2375508 tid 2 2026-02-20T12:01:48.761 INFO:tasks.rados.rados.0.trial153.stdout:581: snap_create 2026-02-20T12:01:48.761 INFO:tasks.rados.rados.0.trial153.stdout:570: finishing write tid 1 to trial15342161-40 2026-02-20T12:01:48.761 INFO:tasks.rados.rados.0.trial153.stdout:569: finishing write tid 3 to trial15342161-41 2026-02-20T12:01:48.761 INFO:tasks.rados.rados.0.trial153.stdout:569: finishing write tid 4 to trial15342161-41 2026-02-20T12:01:48.761 INFO:tasks.rados.rados.0.trial153.stdout:569: oid 41 updating version 0 to 51 2026-02-20T12:01:48.761 INFO:tasks.rados.rados.0.trial153.stdout:569: oid 41 version 51 is already newer than 51 2026-02-20T12:01:48.761 INFO:tasks.rados.rados.0.trial153.stdout:569: oid 41 version 51 is already newer than 49 2026-02-20T12:01:48.761 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 41 v 51 (ObjNum 274 snap 70 seq_num 274) dirty exists 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:569: left oid 41 (ObjNum 274 snap 70 seq_num 274) 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:570: finishing write tid 2 to trial15342161-40 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:571: finishing write tid 2 to trial15342161-4 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:571: finishing write tid 3 to trial15342161-4 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:574: finishing write tid 1 to trial15342161-33 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:571: finishing write tid 4 to trial15342161-4 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:571: oid 4 updating version 0 to 88 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:571: oid 4 updating version 88 to 90 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:571: oid 4 version 90 is already newer than 90 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 4 v 90 (ObjNum 276 snap 70 seq_num 276) dirty exists 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:571: left oid 4 (ObjNum 276 snap 70 seq_num 276) 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:570: finishing write tid 3 to trial15342161-40 2026-02-20T12:01:48.762 INFO:tasks.rados.rados.0.trial153.stdout:570: finishing write tid 4 to trial15342161-40 2026-02-20T12:01:48.763 INFO:tasks.rados.rados.0.trial153.stdout:570: oid 40 updating version 0 to 94 2026-02-20T12:01:48.763 INFO:tasks.rados.rados.0.trial153.stdout:570: oid 40 version 94 is already newer than 94 2026-02-20T12:01:48.763 INFO:tasks.rados.rados.0.trial153.stdout:570: oid 40 version 94 is already newer than 94 2026-02-20T12:01:48.763 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 40 v 94 (ObjNum 275 snap 70 seq_num 275) dirty exists 2026-02-20T12:01:48.763 INFO:tasks.rados.rados.0.trial153.stdout:570: left oid 40 (ObjNum 275 snap 70 seq_num 275) 2026-02-20T12:01:48.763 INFO:tasks.rados.rados.0.trial153.stdout:573: finishing write tid 1 to trial15342161-46 2026-02-20T12:01:48.763 INFO:tasks.rados.rados.0.trial153.stdout:576: expect (ObjNum 196 snap 47 seq_num 196) 2026-02-20T12:01:48.820 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:48.858 INFO:tasks.rados.rados.0.trial153.stdout:575: finishing write tid 1 to trial15342161-42 2026-02-20T12:01:48.858 INFO:tasks.rados.rados.0.trial153.stdout:577: finishing write tid 1 to trial15342161-13 2026-02-20T12:01:48.858 INFO:tasks.rados.rados.0.trial153.stdout:577: finishing write tid 2 to trial15342161-13 2026-02-20T12:01:48.858 INFO:tasks.rados.rados.0.trial153.stdout:574: finishing write tid 2 to trial15342161-33 2026-02-20T12:01:48.858 INFO:tasks.rados.rados.0.trial153.stdout:575: finishing write tid 2 to trial15342161-42 2026-02-20T12:01:48.858 INFO:tasks.rados.rados.0.trial153.stdout:572: finishing write tid 1 to trial15342161-6 2026-02-20T12:01:48.858 INFO:tasks.rados.rados.0.trial153.stdout:577: finishing write tid 3 to trial15342161-13 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:577: finishing write tid 4 to trial15342161-13 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:577: oid 13 updating version 0 to 212 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:577: oid 13 version 212 is already newer than 212 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:577: oid 13 version 212 is already newer than 212 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 13 v 212 (ObjNum 281 snap 70 seq_num 281) dirty exists 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:577: left oid 13 (ObjNum 281 snap 70 seq_num 281) 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:574: finishing write tid 3 to trial15342161-33 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:574: finishing write tid 4 to trial15342161-33 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:574: oid 33 updating version 0 to 66 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:574: oid 33 updating version 66 to 68 2026-02-20T12:01:48.859 INFO:tasks.rados.rados.0.trial153.stdout:574: oid 33 version 68 is already newer than 68 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 33 v 68 (ObjNum 279 snap 70 seq_num 279) dirty exists 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:574: left oid 33 (ObjNum 279 snap 70 seq_num 279) 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:575: finishing write tid 3 to trial15342161-42 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:575: finishing write tid 4 to trial15342161-42 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:575: oid 42 updating version 0 to 54 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:575: oid 42 version 54 is already newer than 54 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:575: oid 42 version 54 is already newer than 52 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 54 (ObjNum 280 snap 70 seq_num 280) dirty exists 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:575: left oid 42 (ObjNum 280 snap 70 seq_num 280) 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:572: finishing write tid 2 to trial15342161-6 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:579: finishing write tid 1 to trial15342161-26 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:572: finishing write tid 3 to trial15342161-6 2026-02-20T12:01:48.860 INFO:tasks.rados.rados.0.trial153.stdout:573: finishing write tid 2 to trial15342161-46 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:572: finishing write tid 4 to trial15342161-6 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:572: oid 6 updating version 0 to 162 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:572: oid 6 updating version 162 to 164 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:572: oid 6 version 164 is already newer than 164 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 6 v 164 (ObjNum 277 snap 70 seq_num 277) dirty exists 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:572: left oid 6 (ObjNum 277 snap 70 seq_num 277) 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:573: finishing write tid 3 to trial15342161-46 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:573: finishing write tid 4 to trial15342161-46 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:573: oid 46 updating version 0 to 25 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:573: oid 46 version 25 is already newer than 25 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:573: oid 46 version 25 is already newer than 23 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 46 v 25 (ObjNum 278 snap 70 seq_num 278) dirty exists 2026-02-20T12:01:48.861 INFO:tasks.rados.rados.0.trial153.stdout:573: left oid 46 (ObjNum 278 snap 70 seq_num 278) 2026-02-20T12:01:48.862 INFO:tasks.rados.rados.0.trial153.stdout:579: finishing write tid 2 to trial15342161-26 2026-02-20T12:01:48.862 INFO:tasks.rados.rados.0.trial153.stdout:578: expect (ObjNum 259 snap 68 seq_num 259) 2026-02-20T12:01:48.983 INFO:tasks.rados.rados.0.trial153.stdout:579: finishing write tid 3 to trial15342161-26 2026-02-20T12:01:48.983 INFO:tasks.rados.rados.0.trial153.stdout:579: finishing write tid 4 to trial15342161-26 2026-02-20T12:01:48.983 INFO:tasks.rados.rados.0.trial153.stdout:579: oid 26 updating version 0 to 144 2026-02-20T12:01:48.983 INFO:tasks.rados.rados.0.trial153.stdout:579: oid 26 version 144 is already newer than 144 2026-02-20T12:01:48.983 INFO:tasks.rados.rados.0.trial153.stdout:579: oid 26 version 144 is already newer than 142 2026-02-20T12:01:48.983 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 26 v 144 (ObjNum 282 snap 70 seq_num 282) dirty exists 2026-02-20T12:01:48.983 INFO:tasks.rados.rados.0.trial153.stdout:579: left oid 26 (ObjNum 282 snap 70 seq_num 282) 2026-02-20T12:01:48.984 INFO:tasks.rados.rados.0.trial153.stdout:580: finishing write tid 1 to trial15342161-12 2026-02-20T12:01:48.984 INFO:tasks.rados.rados.0.trial153.stdout:580: finishing write tid 2 to trial15342161-12 2026-02-20T12:01:48.984 INFO:tasks.rados.rados.0.trial153.stdout:580: finishing write tid 3 to trial15342161-12 2026-02-20T12:01:48.984 INFO:tasks.rados.rados.0.trial153.stdout:580: finishing write tid 4 to trial15342161-12 2026-02-20T12:01:48.984 INFO:tasks.rados.rados.0.trial153.stdout:580: oid 12 updating version 0 to 28 2026-02-20T12:01:48.984 INFO:tasks.rados.rados.0.trial153.stdout:580: oid 12 version 28 is already newer than 28 2026-02-20T12:01:48.984 INFO:tasks.rados.rados.0.trial153.stdout:580: oid 12 version 28 is already newer than 28 2026-02-20T12:01:48.984 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 12 v 28 (ObjNum 283 snap 70 seq_num 283) dirty exists 2026-02-20T12:01:48.984 INFO:tasks.rados.rados.0.trial153.stdout:580: left oid 12 (ObjNum 283 snap 70 seq_num 283) 2026-02-20T12:01:49.547 INFO:tasks.rados.rados.0.trial153.stdout:569: done (12 left) 2026-02-20T12:01:49.547 INFO:tasks.rados.rados.0.trial153.stdout:570: done (11 left) 2026-02-20T12:01:49.547 INFO:tasks.rados.rados.0.trial153.stdout:571: done (10 left) 2026-02-20T12:01:49.547 INFO:tasks.rados.rados.0.trial153.stdout:572: done (9 left) 2026-02-20T12:01:49.547 INFO:tasks.rados.rados.0.trial153.stdout:573: done (8 left) 2026-02-20T12:01:49.547 INFO:tasks.rados.rados.0.trial153.stdout:574: done (7 left) 2026-02-20T12:01:49.547 INFO:tasks.rados.rados.0.trial153.stdout:575: done (6 left) 2026-02-20T12:01:49.547 INFO:tasks.rados.rados.0.trial153.stdout:576: done (5 left) 2026-02-20T12:01:49.548 INFO:tasks.rados.rados.0.trial153.stdout:577: done (4 left) 2026-02-20T12:01:49.548 INFO:tasks.rados.rados.0.trial153.stdout:578: done (3 left) 2026-02-20T12:01:49.548 INFO:tasks.rados.rados.0.trial153.stdout:579: done (2 left) 2026-02-20T12:01:49.548 INFO:tasks.rados.rados.0.trial153.stdout:580: done (1 left) 2026-02-20T12:01:49.548 INFO:tasks.rados.rados.0.trial153.stdout:581: done (0 left) 2026-02-20T12:01:49.548 INFO:tasks.rados.rados.0.trial153.stdout:582: delete oid 31 current snap is 71 2026-02-20T12:01:49.585 INFO:tasks.rados.rados.0.trial153.stdout:582: done (0 left) 2026-02-20T12:01:49.585 INFO:tasks.rados.rados.0.trial153.stdout:583: snap_create 2026-02-20T12:01:49.670 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:49.743 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:49.745 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:49.874 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:49.952 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:49.946+0000 7fb0f08cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:49.955 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:49.951+0000 7fb0f08cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:49.982 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:49.999 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:50.551 INFO:tasks.rados.rados.0.trial153.stdout:583: done (0 left) 2026-02-20T12:01:50.551 INFO:tasks.rados.rados.0.trial153.stdout:584: write oid 17 current snap is 72 2026-02-20T12:01:50.551 INFO:tasks.rados.rados.0.trial153.stdout:584: seq_num 285 ranges {789995=551026,2027721=468420} 2026-02-20T12:01:50.556 INFO:tasks.rados.rados.0.trial153.stdout:584: writing trial15342161-17 from 789995 to 1341021 tid 1 2026-02-20T12:01:50.561 INFO:tasks.rados.rados.0.trial153.stdout:584: writing trial15342161-17 from 2027721 to 2496141 tid 2 2026-02-20T12:01:50.561 INFO:tasks.rados.rados.0.trial153.stdout:585: write oid 21 current snap is 72 2026-02-20T12:01:50.562 INFO:tasks.rados.rados.0.trial153.stdout:585: seq_num 286 ranges {406802=626275,1686203=728431} 2026-02-20T12:01:50.566 INFO:tasks.rados.rados.0.trial153.stdout:585: writing trial15342161-21 from 406802 to 1033077 tid 1 2026-02-20T12:01:50.571 INFO:tasks.rados.rados.0.trial153.stdout:585: writing trial15342161-21 from 1686203 to 2414634 tid 2 2026-02-20T12:01:50.572 INFO:tasks.rados.rados.0.trial153.stdout:586: write (excl) oid 16 current snap is 72 2026-02-20T12:01:50.572 INFO:tasks.rados.rados.0.trial153.stdout:586: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2026-02-20T12:01:50.575 INFO:tasks.rados.rados.0.trial153.stdout:586: writing trial15342161-16 from 423609 to 1125133 tid 1 2026-02-20T12:01:50.579 INFO:tasks.rados.rados.0.trial153.stdout:586: writing trial15342161-16 from 1628332 to 2100421 tid 2 2026-02-20T12:01:50.582 INFO:tasks.rados.rados.0.trial153.stdout:586: writing trial15342161-16 from 2648987 to 2823609 tid 3 2026-02-20T12:01:50.582 INFO:tasks.rados.rados.0.trial153.stdout:587: write (excl) oid 25 current snap is 72 2026-02-20T12:01:50.582 INFO:tasks.rados.rados.0.trial153.stdout:587: seq_num 288 ranges {440416=776773,1686814=732100} 2026-02-20T12:01:50.586 INFO:tasks.rados.rados.0.trial153.stdout:587: writing trial15342161-25 from 440416 to 1217189 tid 1 2026-02-20T12:01:50.591 INFO:tasks.rados.rados.0.trial153.stdout:587: writing trial15342161-25 from 1686814 to 2418914 tid 2 2026-02-20T12:01:50.591 INFO:tasks.rados.rados.0.trial153.stdout:588: write oid 47 current snap is 72 2026-02-20T12:01:50.591 INFO:tasks.rados.rados.0.trial153.stdout:588: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2026-02-20T12:01:50.595 INFO:tasks.rados.rados.0.trial153.stdout:588: writing trial15342161-47 from 457223 to 1025598 tid 1 2026-02-20T12:01:50.598 INFO:tasks.rados.rados.0.trial153.stdout:588: writing trial15342161-47 from 1461649 to 1937407 tid 2 2026-02-20T12:01:50.602 INFO:tasks.rados.rados.0.trial153.stdout:588: writing trial15342161-47 from 2420186 to 2857223 tid 3 2026-02-20T12:01:50.602 INFO:tasks.rados.rados.0.trial153.stdout:589: delete oid 22 current snap is 72 2026-02-20T12:01:50.602 INFO:tasks.rados.rados.0.trial153.stdout:585: finishing write tid 1 to trial15342161-21 2026-02-20T12:01:50.602 INFO:tasks.rados.rados.0.trial153.stdout:584: finishing write tid 1 to trial15342161-17 2026-02-20T12:01:50.602 INFO:tasks.rados.rados.0.trial153.stdout:585: finishing write tid 2 to trial15342161-21 2026-02-20T12:01:50.614 INFO:tasks.rados.rados.0.trial153.stdout:589: done (5 left) 2026-02-20T12:01:50.614 INFO:tasks.rados.rados.0.trial153.stdout:590: delete oid 44 current snap is 72 2026-02-20T12:01:50.616 INFO:tasks.rados.rados.0.trial153.stdout:584: finishing write tid 2 to trial15342161-17 2026-02-20T12:01:50.616 INFO:tasks.rados.rados.0.trial153.stdout:585: finishing write tid 3 to trial15342161-21 2026-02-20T12:01:50.622 INFO:tasks.rados.rados.0.trial153.stdout:585: finishing write tid 4 to trial15342161-21 2026-02-20T12:01:50.622 INFO:tasks.rados.rados.0.trial153.stdout:585: oid 21 updating version 0 to 101 2026-02-20T12:01:50.623 INFO:tasks.rados.rados.0.trial153.stdout:585: oid 21 version 101 is already newer than 101 2026-02-20T12:01:50.623 INFO:tasks.rados.rados.0.trial153.stdout:585: oid 21 version 101 is already newer than 101 2026-02-20T12:01:50.623 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 21 v 101 (ObjNum 286 snap 72 seq_num 286) dirty exists 2026-02-20T12:01:50.623 INFO:tasks.rados.rados.0.trial153.stdout:585: left oid 21 (ObjNum 286 snap 72 seq_num 286) 2026-02-20T12:01:50.628 INFO:tasks.rados.rados.0.trial153.stdout:584: finishing write tid 3 to trial15342161-17 2026-02-20T12:01:50.629 INFO:tasks.rados.rados.0.trial153.stdout:588: finishing write tid 1 to trial15342161-47 2026-02-20T12:01:50.630 INFO:tasks.rados.rados.0.trial153.stdout:584: finishing write tid 4 to trial15342161-17 2026-02-20T12:01:50.630 INFO:tasks.rados.rados.0.trial153.stdout:584: oid 17 updating version 0 to 148 2026-02-20T12:01:50.630 INFO:tasks.rados.rados.0.trial153.stdout:584: oid 17 version 148 is already newer than 148 2026-02-20T12:01:50.630 INFO:tasks.rados.rados.0.trial153.stdout:584: oid 17 version 148 is already newer than 148 2026-02-20T12:01:50.631 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 17 v 148 (ObjNum 285 snap 72 seq_num 285) dirty exists 2026-02-20T12:01:50.631 INFO:tasks.rados.rados.0.trial153.stdout:584: left oid 17 (ObjNum 285 snap 72 seq_num 285) 2026-02-20T12:01:50.636 INFO:tasks.rados.rados.0.trial153.stdout:586: finishing write tid 1 to trial15342161-16 2026-02-20T12:01:50.640 INFO:tasks.rados.rados.0.trial153.stdout:588: finishing write tid 2 to trial15342161-47 2026-02-20T12:01:50.647 INFO:tasks.rados.rados.0.trial153.stdout:588: finishing write tid 3 to trial15342161-47 2026-02-20T12:01:50.652 INFO:tasks.rados.rados.0.trial153.stdout:588: finishing write tid 4 to trial15342161-47 2026-02-20T12:01:50.654 INFO:tasks.rados.rados.0.trial153.stdout:588: finishing write tid 5 to trial15342161-47 2026-02-20T12:01:50.654 INFO:tasks.rados.rados.0.trial153.stdout:588: oid 47 updating version 0 to 99 2026-02-20T12:01:50.654 INFO:tasks.rados.rados.0.trial153.stdout:588: oid 47 version 99 is already newer than 99 2026-02-20T12:01:50.654 INFO:tasks.rados.rados.0.trial153.stdout:588: oid 47 version 99 is already newer than 99 2026-02-20T12:01:50.654 INFO:tasks.rados.rados.0.trial153.stdout:588: oid 47 version 99 is already newer than 99 2026-02-20T12:01:50.654 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 47 v 99 (ObjNum 289 snap 72 seq_num 289) dirty exists 2026-02-20T12:01:50.654 INFO:tasks.rados.rados.0.trial153.stdout:588: left oid 47 (ObjNum 289 snap 72 seq_num 289) 2026-02-20T12:01:50.658 INFO:tasks.rados.rados.0.trial153.stdout:587: finishing write tid 1 to trial15342161-25 2026-02-20T12:01:50.660 INFO:tasks.rados.rados.0.trial153.stdout:586: finishing write tid 2 to trial15342161-16 2026-02-20T12:01:50.674 INFO:tasks.rados.rados.0.trial153.stdout:586: finishing write tid 3 to trial15342161-16 2026-02-20T12:01:50.681 INFO:tasks.rados.rados.0.trial153.stdout:587: finishing write tid 2 to trial15342161-25 2026-02-20T12:01:50.690 INFO:tasks.rados.rados.0.trial153.stdout:586: finishing write tid 4 to trial15342161-16 2026-02-20T12:01:50.694 INFO:tasks.rados.rados.0.trial153.stdout:587: finishing write tid 3 to trial15342161-25 2026-02-20T12:01:50.694 INFO:tasks.rados.rados.0.trial153.stdout:586: finishing write tid 5 to trial15342161-16 2026-02-20T12:01:50.694 INFO:tasks.rados.rados.0.trial153.stdout:586: oid 16 updating version 0 to 105 2026-02-20T12:01:50.694 INFO:tasks.rados.rados.0.trial153.stdout:586: oid 16 version 105 is already newer than 105 2026-02-20T12:01:50.694 INFO:tasks.rados.rados.0.trial153.stdout:586: oid 16 version 105 is already newer than 102 2026-02-20T12:01:50.694 INFO:tasks.rados.rados.0.trial153.stdout:586: oid 16 version 105 is already newer than 105 2026-02-20T12:01:50.694 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 16 v 105 (ObjNum 287 snap 72 seq_num 287) dirty exists 2026-02-20T12:01:50.695 INFO:tasks.rados.rados.0.trial153.stdout:586: left oid 16 (ObjNum 287 snap 72 seq_num 287) 2026-02-20T12:01:50.696 INFO:tasks.rados.rados.0.trial153.stdout:587: finishing write tid 4 to trial15342161-25 2026-02-20T12:01:50.696 INFO:tasks.rados.rados.0.trial153.stdout:587: oid 25 updating version 0 to 48 2026-02-20T12:01:50.696 INFO:tasks.rados.rados.0.trial153.stdout:587: oid 25 version 48 is already newer than 48 2026-02-20T12:01:50.696 INFO:tasks.rados.rados.0.trial153.stdout:587: oid 25 version 48 is already newer than 46 2026-02-20T12:01:50.696 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 25 v 48 (ObjNum 288 snap 72 seq_num 288) dirty exists 2026-02-20T12:01:50.697 INFO:tasks.rados.rados.0.trial153.stdout:587: left oid 25 (ObjNum 288 snap 72 seq_num 288) 2026-02-20T12:01:50.709 INFO:tasks.rados.rados.0.trial153.stdout:584: done (5 left) 2026-02-20T12:01:50.709 INFO:tasks.rados.rados.0.trial153.stdout:585: done (4 left) 2026-02-20T12:01:50.709 INFO:tasks.rados.rados.0.trial153.stdout:586: done (3 left) 2026-02-20T12:01:50.709 INFO:tasks.rados.rados.0.trial153.stdout:587: done (2 left) 2026-02-20T12:01:50.709 INFO:tasks.rados.rados.0.trial153.stdout:588: done (1 left) 2026-02-20T12:01:50.709 INFO:tasks.rados.rados.0.trial153.stdout:590: done (0 left) 2026-02-20T12:01:50.709 INFO:tasks.rados.rados.0.trial153.stdout:591: delete oid 10 current snap is 72 2026-02-20T12:01:50.734 INFO:tasks.rados.rados.0.trial153.stdout:591: done (0 left) 2026-02-20T12:01:50.734 INFO:tasks.rados.rados.0.trial153.stdout:592: read oid 36 snap 70 2026-02-20T12:01:50.734 INFO:tasks.rados.rados.0.trial153.stdout:592: expect deleted 2026-02-20T12:01:50.734 INFO:tasks.rados.rados.0.trial153.stdout:593: read oid 5 snap -1 2026-02-20T12:01:50.734 INFO:tasks.rados.rados.0.trial153.stdout:593: expect deleted 2026-02-20T12:01:50.734 INFO:tasks.rados.rados.0.trial153.stdout:594: read oid 24 snap -1 2026-02-20T12:01:50.735 INFO:tasks.rados.rados.0.trial153.stdout:594: expect (ObjNum 248 snap 63 seq_num 248) 2026-02-20T12:01:50.735 INFO:tasks.rados.rados.0.trial153.stdout:595: delete oid 31 current snap is 72 2026-02-20T12:01:50.739 INFO:tasks.rados.rados.0.trial153.stdout:593: done (3 left) 2026-02-20T12:01:50.739 INFO:tasks.rados.rados.0.trial153.stdout:595: done (2 left) 2026-02-20T12:01:50.739 INFO:tasks.rados.rados.0.trial153.stdout:596: write oid 23 current snap is 72 2026-02-20T12:01:50.739 INFO:tasks.rados.rados.0.trial153.stdout:596: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2026-02-20T12:01:50.743 INFO:tasks.rados.rados.0.trial153.stdout:596: writing trial15342161-23 from 541258 to 1085878 tid 1 2026-02-20T12:01:50.748 INFO:tasks.rados.rados.0.trial153.stdout:596: writing trial15342161-23 from 1637706 to 2264460 tid 2 2026-02-20T12:01:50.751 INFO:tasks.rados.rados.0.trial153.stdout:596: writing trial15342161-23 from 2840948 to 2941258 tid 3 2026-02-20T12:01:50.751 INFO:tasks.rados.rados.0.trial153.stdout:597: snap_create 2026-02-20T12:01:50.752 INFO:tasks.rados.rados.0.trial153.stdout:594: expect (ObjNum 248 snap 63 seq_num 248) 2026-02-20T12:01:50.800 INFO:tasks.rados.rados.0.trial153.stdout:596: finishing write tid 1 to trial15342161-23 2026-02-20T12:01:50.801 INFO:tasks.rados.rados.0.trial153.stdout:596: finishing write tid 2 to trial15342161-23 2026-02-20T12:01:50.801 INFO:tasks.rados.rados.0.trial153.stdout:596: finishing write tid 3 to trial15342161-23 2026-02-20T12:01:50.801 INFO:tasks.rados.rados.0.trial153.stdout:596: finishing write tid 4 to trial15342161-23 2026-02-20T12:01:50.801 INFO:tasks.rados.rados.0.trial153.stdout:596: finishing write tid 5 to trial15342161-23 2026-02-20T12:01:50.801 INFO:tasks.rados.rados.0.trial153.stdout:596: oid 23 updating version 0 to 103 2026-02-20T12:01:50.801 INFO:tasks.rados.rados.0.trial153.stdout:596: oid 23 version 103 is already newer than 103 2026-02-20T12:01:50.801 INFO:tasks.rados.rados.0.trial153.stdout:596: oid 23 version 103 is already newer than 103 2026-02-20T12:01:50.801 INFO:tasks.rados.rados.0.trial153.stdout:596: oid 23 version 103 is already newer than 103 2026-02-20T12:01:50.801 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 23 v 103 (ObjNum 294 snap 72 seq_num 294) dirty exists 2026-02-20T12:01:50.801 INFO:tasks.rados.rados.0.trial153.stdout:596: left oid 23 (ObjNum 294 snap 72 seq_num 294) 2026-02-20T12:01:51.555 INFO:tasks.rados.rados.0.trial153.stdout:592: done (3 left) 2026-02-20T12:01:51.556 INFO:tasks.rados.rados.0.trial153.stdout:594: done (2 left) 2026-02-20T12:01:51.556 INFO:tasks.rados.rados.0.trial153.stdout:596: done (1 left) 2026-02-20T12:01:51.556 INFO:tasks.rados.rados.0.trial153.stdout:597: done (0 left) 2026-02-20T12:01:51.556 INFO:tasks.rados.rados.0.trial153.stdout:598: read oid 42 snap -1 2026-02-20T12:01:51.556 INFO:tasks.rados.rados.0.trial153.stdout:598: expect (ObjNum 280 snap 70 seq_num 280) 2026-02-20T12:01:51.556 INFO:tasks.rados.rados.0.trial153.stdout:599: snap_remove snap 71 2026-02-20T12:01:52.000 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:52.077 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:52.071+0000 7fdce765a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:52.080 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:52.075+0000 7fdce63d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:52.106 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:52.120 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:52.557 INFO:tasks.rados.rados.0.trial153.stdout:599: done (1 left) 2026-02-20T12:01:52.557 INFO:tasks.rados.rados.0.trial153.stdout:600: delete oid 35 current snap is 73 2026-02-20T12:01:52.557 INFO:tasks.rados.rados.0.trial153.stdout:598: expect (ObjNum 280 snap 70 seq_num 280) 2026-02-20T12:01:52.557 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:52.602 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:52.626 INFO:tasks.rados.rados.0.trial153.stdout:598: done (1 left) 2026-02-20T12:01:52.626 INFO:tasks.rados.rados.0.trial153.stdout:600: done (0 left) 2026-02-20T12:01:52.626 INFO:tasks.rados.rados.0.trial153.stdout:601: delete oid 37 current snap is 73 2026-02-20T12:01:52.628 INFO:tasks.rados.rados.0.trial153.stdout:601: done (0 left) 2026-02-20T12:01:52.628 INFO:tasks.rados.rados.0.trial153.stdout:602: delete oid 15 current snap is 73 2026-02-20T12:01:52.652 INFO:tasks.rados.rados.0.trial153.stdout:602: done (0 left) 2026-02-20T12:01:52.652 INFO:tasks.rados.rados.0.trial153.stdout:603: read oid 44 snap -1 2026-02-20T12:01:52.652 INFO:tasks.rados.rados.0.trial153.stdout:603: expect deleted 2026-02-20T12:01:52.652 INFO:tasks.rados.rados.0.trial153.stdout:604: snap_remove snap 70 2026-02-20T12:01:53.427 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~160s 2026-02-20T12:01:53.563 INFO:tasks.rados.rados.0.trial153.stdout:604: done (1 left) 2026-02-20T12:01:53.563 INFO:tasks.rados.rados.0.trial153.stdout:605: read oid 20 snap -1 2026-02-20T12:01:53.563 INFO:tasks.rados.rados.0.trial153.stdout:605: expect (ObjNum 245 snap 61 seq_num 245) 2026-02-20T12:01:53.563 INFO:tasks.rados.rados.0.trial153.stdout:606: snap_create 2026-02-20T12:01:53.597 INFO:tasks.rados.rados.0.trial153.stdout:605: expect (ObjNum 245 snap 61 seq_num 245) 2026-02-20T12:01:53.627 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:54.121 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:54.199 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:54.193+0000 7f3fe8ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:54.202 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:54.197+0000 7f3fe8ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:54.228 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:54.245 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:54.565 INFO:tasks.rados.rados.0.trial153.stdout:603: done (2 left) 2026-02-20T12:01:54.565 INFO:tasks.rados.rados.0.trial153.stdout:605: done (1 left) 2026-02-20T12:01:54.565 INFO:tasks.rados.rados.0.trial153.stdout:606: done (0 left) 2026-02-20T12:01:54.565 INFO:tasks.rados.rados.0.trial153.stdout:607: snap_remove snap 72 2026-02-20T12:01:54.747 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:54.818 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:54.821 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:01:55.569 INFO:tasks.rados.rados.0.trial153.stdout:607: done (0 left) 2026-02-20T12:01:55.570 INFO:tasks.rados.rados.0.trial153.stdout:608: write (excl) oid 28 current snap is 74 2026-02-20T12:01:55.570 INFO:tasks.rados.rados.0.trial153.stdout:608: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2026-02-20T12:01:55.574 INFO:tasks.rados.rados.0.trial153.stdout:608: writing trial15342161-28 from 608486 to 1170455 tid 1 2026-02-20T12:01:55.579 INFO:tasks.rados.rados.0.trial153.stdout:608: writing trial15342161-28 from 1871634 to 2505726 tid 2 2026-02-20T12:01:55.581 INFO:tasks.rados.rados.0.trial153.stdout:608: writing trial15342161-28 from 2950640 to 3008486 tid 3 2026-02-20T12:01:55.581 INFO:tasks.rados.rados.0.trial153.stdout:609: snap_create 2026-02-20T12:01:55.652 INFO:tasks.rados.rados.0.trial153.stdout:608: finishing write tid 1 to trial15342161-28 2026-02-20T12:01:55.667 INFO:tasks.rados.rados.0.trial153.stdout:608: finishing write tid 2 to trial15342161-28 2026-02-20T12:01:55.673 INFO:tasks.rados.rados.0.trial153.stdout:608: finishing write tid 3 to trial15342161-28 2026-02-20T12:01:55.679 INFO:tasks.rados.rados.0.trial153.stdout:608: finishing write tid 4 to trial15342161-28 2026-02-20T12:01:55.680 INFO:tasks.rados.rados.0.trial153.stdout:608: finishing write tid 5 to trial15342161-28 2026-02-20T12:01:55.681 INFO:tasks.rados.rados.0.trial153.stdout:608: oid 28 updating version 0 to 110 2026-02-20T12:01:55.681 INFO:tasks.rados.rados.0.trial153.stdout:608: oid 28 version 110 is already newer than 107 2026-02-20T12:01:55.681 INFO:tasks.rados.rados.0.trial153.stdout:608: oid 28 version 110 is already newer than 110 2026-02-20T12:01:55.681 INFO:tasks.rados.rados.0.trial153.stdout:608: oid 28 version 110 is already newer than 110 2026-02-20T12:01:55.681 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 28 v 110 (ObjNum 298 snap 74 seq_num 298) dirty exists 2026-02-20T12:01:55.681 INFO:tasks.rados.rados.0.trial153.stdout:608: left oid 28 (ObjNum 298 snap 74 seq_num 298) 2026-02-20T12:01:56.246 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:01:56.323 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:56.318+0000 7ff319ffd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:56.326 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:56.322+0000 7ff319ffd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:56.352 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:56.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:56.513 INFO:tasks.rados.rados.0.trial153.stdout:608: done (1 left) 2026-02-20T12:01:56.514 INFO:tasks.rados.rados.0.trial153.stdout:609: done (0 left) 2026-02-20T12:01:56.514 INFO:tasks.rados.rados.0.trial153.stdout:610: delete oid 11 current snap is 75 2026-02-20T12:01:56.549 INFO:tasks.rados.rados.0.trial153.stdout:610: done (0 left) 2026-02-20T12:01:56.549 INFO:tasks.rados.rados.0.trial153.stdout:611: snap_remove snap 73 2026-02-20T12:01:57.515 INFO:tasks.rados.rados.0.trial153.stdout:611: done (0 left) 2026-02-20T12:01:57.515 INFO:tasks.rados.rados.0.trial153.stdout:612: read oid 44 snap -1 2026-02-20T12:01:57.515 INFO:tasks.rados.rados.0.trial153.stdout:612: expect deleted 2026-02-20T12:01:57.515 INFO:tasks.rados.rados.0.trial153.stdout:613: delete oid 45 current snap is 75 2026-02-20T12:01:57.555 INFO:tasks.rados.rados.0.trial153.stdout:612: done (1 left) 2026-02-20T12:01:57.555 INFO:tasks.rados.rados.0.trial153.stdout:613: done (0 left) 2026-02-20T12:01:57.555 INFO:tasks.rados.rados.0.trial153.stdout:614: write (excl) oid 8 current snap is 75 2026-02-20T12:01:57.555 INFO:tasks.rados.rados.0.trial153.stdout:614: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2026-02-20T12:01:57.560 INFO:tasks.rados.rados.0.trial153.stdout:614: writing trial15342161-8 from 658907 to 1446623 tid 1 2026-02-20T12:01:57.564 INFO:tasks.rados.rados.0.trial153.stdout:614: writing trial15342161-8 from 1930727 to 2428499 tid 2 2026-02-20T12:01:57.567 INFO:tasks.rados.rados.0.trial153.stdout:614: writing trial15342161-8 from 3032909 to 3058907 tid 3 2026-02-20T12:01:57.567 INFO:tasks.rados.rados.0.trial153.stdout:615: write oid 49 current snap is 75 2026-02-20T12:01:57.567 INFO:tasks.rados.rados.0.trial153.stdout:615: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2026-02-20T12:01:57.571 INFO:tasks.rados.rados.0.trial153.stdout:615: writing trial15342161-49 from 675714 to 1138679 tid 1 2026-02-20T12:01:57.575 INFO:tasks.rados.rados.0.trial153.stdout:615: writing trial15342161-49 from 1589209 to 2230639 tid 2 2026-02-20T12:01:57.579 INFO:tasks.rados.rados.0.trial153.stdout:615: writing trial15342161-49 from 2943979 to 3075714 tid 3 2026-02-20T12:01:57.580 INFO:tasks.rados.rados.0.trial153.stdout:616: read oid 38 snap -1 2026-02-20T12:01:57.580 INFO:tasks.rados.rados.0.trial153.stdout:616: expect (ObjNum 166 snap 36 seq_num 166) 2026-02-20T12:01:57.580 INFO:tasks.rados.rados.0.trial153.stdout:617: read oid 43 snap -1 2026-02-20T12:01:57.580 INFO:tasks.rados.rados.0.trial153.stdout:617: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:01:57.580 INFO:tasks.rados.rados.0.trial153.stdout:618: write oid 14 current snap is 75 2026-02-20T12:01:57.580 INFO:tasks.rados.rados.0.trial153.stdout:618: seq_num 303 ranges {692521=538214,1931338=785088} 2026-02-20T12:01:57.584 INFO:tasks.rados.rados.0.trial153.stdout:618: writing trial15342161-14 from 692521 to 1230735 tid 1 2026-02-20T12:01:57.590 INFO:tasks.rados.rados.0.trial153.stdout:618: writing trial15342161-14 from 1931338 to 2716426 tid 2 2026-02-20T12:01:57.590 INFO:tasks.rados.rados.0.trial153.stdout:619: read oid 18 snap -1 2026-02-20T12:01:57.590 INFO:tasks.rados.rados.0.trial153.stdout:619: expect deleted 2026-02-20T12:01:57.590 INFO:tasks.rados.rados.0.trial153.stdout:620: delete oid 17 current snap is 75 2026-02-20T12:01:57.598 INFO:tasks.rados.rados.0.trial153.stdout:616: expect (ObjNum 166 snap 36 seq_num 166) 2026-02-20T12:01:57.651 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2172416 2026-02-20T12:01:57.679 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2428834 2026-02-20T12:01:57.689 INFO:tasks.rados.rados.0.trial153.stdout:616: done (6 left) 2026-02-20T12:01:57.690 INFO:tasks.rados.rados.0.trial153.stdout:619: done (5 left) 2026-02-20T12:01:57.690 INFO:tasks.rados.rados.0.trial153.stdout:620: done (4 left) 2026-02-20T12:01:57.690 INFO:tasks.rados.rados.0.trial153.stdout:621: delete oid 37 current snap is 75 2026-02-20T12:01:57.690 INFO:tasks.rados.rados.0.trial153.stdout:614: finishing write tid 1 to trial15342161-8 2026-02-20T12:01:57.690 INFO:tasks.rados.rados.0.trial153.stdout:615: finishing write tid 1 to trial15342161-49 2026-02-20T12:01:57.690 INFO:tasks.rados.rados.0.trial153.stdout:615: finishing write tid 2 to trial15342161-49 2026-02-20T12:01:57.691 INFO:tasks.rados.rados.0.trial153.stdout:617: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:01:57.695 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1101161 2026-02-20T12:01:57.818 INFO:tasks.rados.rados.0.trial153.stdout:614: finishing write tid 2 to trial15342161-8 2026-02-20T12:01:57.819 INFO:tasks.rados.rados.0.trial153.stdout:615: finishing write tid 3 to trial15342161-49 2026-02-20T12:01:57.819 INFO:tasks.rados.rados.0.trial153.stdout:614: finishing write tid 3 to trial15342161-8 2026-02-20T12:01:57.819 INFO:tasks.rados.rados.0.trial153.stdout:615: finishing write tid 4 to trial15342161-49 2026-02-20T12:01:57.819 INFO:tasks.rados.rados.0.trial153.stdout:615: finishing write tid 5 to trial15342161-49 2026-02-20T12:01:57.819 INFO:tasks.rados.rados.0.trial153.stdout:615: oid 49 updating version 0 to 157 2026-02-20T12:01:57.819 INFO:tasks.rados.rados.0.trial153.stdout:615: oid 49 version 157 is already newer than 157 2026-02-20T12:01:57.819 INFO:tasks.rados.rados.0.trial153.stdout:615: oid 49 version 157 is already newer than 157 2026-02-20T12:01:57.819 INFO:tasks.rados.rados.0.trial153.stdout:615: oid 49 version 157 is already newer than 157 2026-02-20T12:01:57.819 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 49 v 157 (ObjNum 302 snap 75 seq_num 302) dirty exists 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:615: left oid 49 (ObjNum 302 snap 75 seq_num 302) 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:614: finishing write tid 4 to trial15342161-8 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:615: done (4 left) 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:617: done (3 left) 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:621: done (2 left) 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:622: delete oid 44 current snap is 75 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:618: finishing write tid 1 to trial15342161-14 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:614: finishing write tid 5 to trial15342161-8 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:614: oid 8 updating version 0 to 56 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:614: oid 8 updating version 56 to 59 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:614: oid 8 version 59 is already newer than 59 2026-02-20T12:01:57.820 INFO:tasks.rados.rados.0.trial153.stdout:614: oid 8 version 59 is already newer than 59 2026-02-20T12:01:57.821 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 8 v 59 (ObjNum 301 snap 75 seq_num 301) dirty exists 2026-02-20T12:01:57.821 INFO:tasks.rados.rados.0.trial153.stdout:614: left oid 8 (ObjNum 301 snap 75 seq_num 301) 2026-02-20T12:01:57.821 INFO:tasks.rados.rados.0.trial153.stdout:618: finishing write tid 2 to trial15342161-14 2026-02-20T12:01:57.821 INFO:tasks.rados.rados.0.trial153.stdout:618: finishing write tid 3 to trial15342161-14 2026-02-20T12:01:57.821 INFO:tasks.rados.rados.0.trial153.stdout:618: finishing write tid 4 to trial15342161-14 2026-02-20T12:01:57.821 INFO:tasks.rados.rados.0.trial153.stdout:618: oid 14 updating version 0 to 176 2026-02-20T12:01:57.821 INFO:tasks.rados.rados.0.trial153.stdout:618: oid 14 version 176 is already newer than 176 2026-02-20T12:01:57.821 INFO:tasks.rados.rados.0.trial153.stdout:618: oid 14 version 176 is already newer than 176 2026-02-20T12:01:57.821 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 176 (ObjNum 303 snap 75 seq_num 303) dirty exists 2026-02-20T12:01:57.821 INFO:tasks.rados.rados.0.trial153.stdout:618: left oid 14 (ObjNum 303 snap 75 seq_num 303) 2026-02-20T12:01:57.822 INFO:tasks.rados.rados.0.trial153.stdout:614: done (2 left) 2026-02-20T12:01:57.822 INFO:tasks.rados.rados.0.trial153.stdout:618: done (1 left) 2026-02-20T12:01:57.822 INFO:tasks.rados.rados.0.trial153.stdout:622: done (0 left) 2026-02-20T12:01:57.822 INFO:tasks.rados.rados.0.trial153.stdout:623: delete oid 39 current snap is 75 2026-02-20T12:01:57.828 INFO:tasks.rados.rados.0.trial153.stdout:623: done (0 left) 2026-02-20T12:01:57.828 INFO:tasks.rados.rados.0.trial153.stdout:624: snap_remove snap 74 2026-02-20T12:01:58.370 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:01:58.449 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:58.443+0000 7efc2e9d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:58.452 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:01:58.447+0000 7efc2e9d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:01:58.479 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:01:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:01:58.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:01:58.574 INFO:tasks.rados.rados.0.trial153.stdout:624: done (0 left) 2026-02-20T12:01:58.574 INFO:tasks.rados.rados.0.trial153.stdout:625: read oid 6 snap -1 2026-02-20T12:01:58.574 INFO:tasks.rados.rados.0.trial153.stdout:625: expect (ObjNum 277 snap 70 seq_num 277) 2026-02-20T12:01:58.574 INFO:tasks.rados.rados.0.trial153.stdout:626: read oid 35 snap -1 2026-02-20T12:01:58.574 INFO:tasks.rados.rados.0.trial153.stdout:626: expect deleted 2026-02-20T12:01:58.574 INFO:tasks.rados.rados.0.trial153.stdout:627: write oid 48 current snap is 75 2026-02-20T12:01:58.574 INFO:tasks.rados.rados.0.trial153.stdout:627: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2026-02-20T12:01:58.579 INFO:tasks.rados.rados.0.trial153.stdout:627: writing trial15342161-48 from 776556 to 1407368 tid 1 2026-02-20T12:01:58.583 INFO:tasks.rados.rados.0.trial153.stdout:627: writing trial15342161-48 from 1823748 to 2476185 tid 2 2026-02-20T12:01:58.586 INFO:tasks.rados.rados.0.trial153.stdout:627: writing trial15342161-48 from 3108517 to 3176556 tid 3 2026-02-20T12:01:58.586 INFO:tasks.rados.rados.0.trial153.stdout:628: read oid 46 snap -1 2026-02-20T12:01:58.586 INFO:tasks.rados.rados.0.trial153.stdout:628: expect (ObjNum 278 snap 70 seq_num 278) 2026-02-20T12:01:58.586 INFO:tasks.rados.rados.0.trial153.stdout:629: snap_create 2026-02-20T12:01:58.620 INFO:tasks.rados.rados.0.trial153.stdout:625: expect (ObjNum 277 snap 70 seq_num 277) 2026-02-20T12:01:58.620 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:58.678 INFO:tasks.rados.rados.0.trial153.stdout:627: finishing write tid 1 to trial15342161-48 2026-02-20T12:01:58.678 INFO:tasks.rados.rados.0.trial153.stdout:627: finishing write tid 2 to trial15342161-48 2026-02-20T12:01:58.678 INFO:tasks.rados.rados.0.trial153.stdout:628: expect (ObjNum 278 snap 70 seq_num 278) 2026-02-20T12:01:58.714 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:01:58.787 INFO:tasks.rados.rados.0.trial153.stdout:627: finishing write tid 3 to trial15342161-48 2026-02-20T12:01:58.787 INFO:tasks.rados.rados.0.trial153.stdout:627: finishing write tid 4 to trial15342161-48 2026-02-20T12:01:58.787 INFO:tasks.rados.rados.0.trial153.stdout:627: finishing write tid 5 to trial15342161-48 2026-02-20T12:01:58.787 INFO:tasks.rados.rados.0.trial153.stdout:627: oid 48 updating version 0 to 52 2026-02-20T12:01:58.787 INFO:tasks.rados.rados.0.trial153.stdout:627: oid 48 version 52 is already newer than 52 2026-02-20T12:01:58.787 INFO:tasks.rados.rados.0.trial153.stdout:627: oid 48 version 52 is already newer than 52 2026-02-20T12:01:58.787 INFO:tasks.rados.rados.0.trial153.stdout:627: oid 48 version 52 is already newer than 52 2026-02-20T12:01:58.788 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 48 v 52 (ObjNum 308 snap 75 seq_num 308) dirty exists 2026-02-20T12:01:58.788 INFO:tasks.rados.rados.0.trial153.stdout:627: left oid 48 (ObjNum 308 snap 75 seq_num 308) 2026-02-20T12:01:58.931 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~165s 2026-02-20T12:01:59.575 INFO:tasks.rados.rados.0.trial153.stdout:625: done (4 left) 2026-02-20T12:01:59.576 INFO:tasks.rados.rados.0.trial153.stdout:626: done (3 left) 2026-02-20T12:01:59.576 INFO:tasks.rados.rados.0.trial153.stdout:627: done (2 left) 2026-02-20T12:01:59.576 INFO:tasks.rados.rados.0.trial153.stdout:628: done (1 left) 2026-02-20T12:01:59.576 INFO:tasks.rados.rados.0.trial153.stdout:629: done (0 left) 2026-02-20T12:01:59.576 INFO:tasks.rados.rados.0.trial153.stdout:630: snap_create 2026-02-20T12:01:59.822 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:01:59.894 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:01:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:01:59.897 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:00.497 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:00.576 INFO:tasks.rados.rados.0.trial153.stdout:630: done (0 left) 2026-02-20T12:02:00.577 INFO:tasks.rados.rados.0.trial153.stdout:631: read oid 48 snap 76 2026-02-20T12:02:00.577 INFO:tasks.rados.rados.0.trial153.stdout:631: expect (ObjNum 308 snap 75 seq_num 308) 2026-02-20T12:02:00.577 INFO:tasks.rados.rados.0.trial153.stdout:632: snap_remove snap 75 2026-02-20T12:02:00.579 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:00.573+0000 7f88475e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:00.581 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:00.577+0000 7f88475e4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:00.621 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:00.638 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:01.581 INFO:tasks.rados.rados.0.trial153.stdout:632: done (1 left) 2026-02-20T12:02:01.581 INFO:tasks.rados.rados.0.trial153.stdout:633: write (excl) oid 4 current snap is 77 2026-02-20T12:02:01.581 INFO:tasks.rados.rados.0.trial153.stdout:633: seq_num 309 ranges {793363=706061,2282230=796095} 2026-02-20T12:02:01.587 INFO:tasks.rados.rados.0.trial153.stdout:633: writing trial15342161-4 from 793363 to 1499424 tid 1 2026-02-20T12:02:01.594 INFO:tasks.rados.rados.0.trial153.stdout:633: writing trial15342161-4 from 2282230 to 3078325 tid 2 2026-02-20T12:02:01.594 INFO:tasks.rados.rados.0.trial153.stdout:634: read oid 17 snap 76 2026-02-20T12:02:01.594 INFO:tasks.rados.rados.0.trial153.stdout:634: expect deleted 2026-02-20T12:02:01.594 INFO:tasks.rados.rados.0.trial153.stdout:635: write oid 32 current snap is 77 2026-02-20T12:02:01.594 INFO:tasks.rados.rados.0.trial153.stdout:635: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2026-02-20T12:02:01.598 INFO:tasks.rados.rados.0.trial153.stdout:635: writing trial15342161-32 from 410170 to 1191480 tid 1 2026-02-20T12:02:01.604 INFO:tasks.rados.rados.0.trial153.stdout:635: writing trial15342161-32 from 1940712 to 2596818 tid 2 2026-02-20T12:02:01.606 INFO:tasks.rados.rados.0.trial153.stdout:635: writing trial15342161-32 from 3163363 to 3210170 tid 3 2026-02-20T12:02:01.606 INFO:tasks.rados.rados.0.trial153.stdout:636: read oid 42 snap -1 2026-02-20T12:02:01.606 INFO:tasks.rados.rados.0.trial153.stdout:636: expect (ObjNum 280 snap 70 seq_num 280) 2026-02-20T12:02:01.606 INFO:tasks.rados.rados.0.trial153.stdout:637: write (excl) oid 25 current snap is 77 2026-02-20T12:02:01.607 INFO:tasks.rados.rados.0.trial153.stdout:637: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2026-02-20T12:02:01.609 INFO:tasks.rados.rados.0.trial153.stdout:637: writing trial15342161-25 from 426977 to 883536 tid 1 2026-02-20T12:02:01.615 INFO:tasks.rados.rados.0.trial153.stdout:637: writing trial15342161-25 from 1482841 to 2282605 tid 2 2026-02-20T12:02:01.618 INFO:tasks.rados.rados.0.trial153.stdout:637: writing trial15342161-25 from 2958080 to 3226977 tid 3 2026-02-20T12:02:01.618 INFO:tasks.rados.rados.0.trial153.stdout:638: write (excl) oid 5 current snap is 77 2026-02-20T12:02:01.619 INFO:tasks.rados.rados.0.trial153.stdout:631: expect (ObjNum 308 snap 75 seq_num 308) 2026-02-20T12:02:01.681 INFO:tasks.rados.rados.0.trial153.stdout:638: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2026-02-20T12:02:01.686 INFO:tasks.rados.rados.0.trial153.stdout:638: writing trial15342161-5 from 443784 to 1091945 tid 1 2026-02-20T12:02:01.691 INFO:tasks.rados.rados.0.trial153.stdout:638: writing trial15342161-5 from 1657676 to 2317451 tid 2 2026-02-20T12:02:01.695 INFO:tasks.rados.rados.0.trial153.stdout:638: writing trial15342161-5 from 2818209 to 3243784 tid 3 2026-02-20T12:02:01.695 INFO:tasks.rados.rados.0.trial153.stdout:631: done (6 left) 2026-02-20T12:02:01.695 INFO:tasks.rados.rados.0.trial153.stdout:639: snap_create 2026-02-20T12:02:01.695 INFO:tasks.rados.rados.0.trial153.stdout:636: expect (ObjNum 280 snap 70 seq_num 280) 2026-02-20T12:02:01.695 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 184303 2026-02-20T12:02:01.763 INFO:tasks.rados.rados.0.trial153.stdout:633: finishing write tid 1 to trial15342161-4 2026-02-20T12:02:01.763 INFO:tasks.rados.rados.0.trial153.stdout:637: finishing write tid 1 to trial15342161-25 2026-02-20T12:02:01.763 INFO:tasks.rados.rados.0.trial153.stdout:633: finishing write tid 2 to trial15342161-4 2026-02-20T12:02:01.763 INFO:tasks.rados.rados.0.trial153.stdout:637: finishing write tid 2 to trial15342161-25 2026-02-20T12:02:01.763 INFO:tasks.rados.rados.0.trial153.stdout:633: finishing write tid 3 to trial15342161-4 2026-02-20T12:02:01.763 INFO:tasks.rados.rados.0.trial153.stdout:633: finishing write tid 4 to trial15342161-4 2026-02-20T12:02:01.763 INFO:tasks.rados.rados.0.trial153.stdout:633: oid 4 updating version 0 to 92 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:633: oid 4 updating version 92 to 94 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:633: oid 4 version 94 is already newer than 94 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 4 v 94 (ObjNum 309 snap 77 seq_num 309) dirty exists 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:633: left oid 4 (ObjNum 309 snap 77 seq_num 309) 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:637: finishing write tid 3 to trial15342161-25 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:637: finishing write tid 4 to trial15342161-25 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:637: finishing write tid 5 to trial15342161-25 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:637: oid 25 updating version 0 to 53 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:637: oid 25 version 53 is already newer than 53 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:637: oid 25 version 53 is already newer than 53 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:637: oid 25 version 53 is already newer than 50 2026-02-20T12:02:01.764 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 25 v 53 (ObjNum 311 snap 77 seq_num 311) dirty exists 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:637: left oid 25 (ObjNum 311 snap 77 seq_num 311) 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:635: finishing write tid 1 to trial15342161-32 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:635: finishing write tid 2 to trial15342161-32 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:635: finishing write tid 3 to trial15342161-32 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:635: finishing write tid 4 to trial15342161-32 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:635: finishing write tid 5 to trial15342161-32 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:635: oid 32 updating version 0 to 99 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:635: oid 32 version 99 is already newer than 99 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:635: oid 32 version 99 is already newer than 99 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:635: oid 32 version 99 is already newer than 99 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 32 v 99 (ObjNum 310 snap 77 seq_num 310) dirty exists 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:635: left oid 32 (ObjNum 310 snap 77 seq_num 310) 2026-02-20T12:02:01.765 INFO:tasks.rados.rados.0.trial153.stdout:638: finishing write tid 1 to trial15342161-5 2026-02-20T12:02:01.766 INFO:tasks.rados.rados.0.trial153.stdout:638: finishing write tid 2 to trial15342161-5 2026-02-20T12:02:01.766 INFO:tasks.rados.rados.0.trial153.stdout:638: finishing write tid 3 to trial15342161-5 2026-02-20T12:02:01.766 INFO:tasks.rados.rados.0.trial153.stdout:638: finishing write tid 4 to trial15342161-5 2026-02-20T12:02:01.766 INFO:tasks.rados.rados.0.trial153.stdout:638: finishing write tid 5 to trial15342161-5 2026-02-20T12:02:01.766 INFO:tasks.rados.rados.0.trial153.stdout:638: oid 5 updating version 0 to 63 2026-02-20T12:02:01.766 INFO:tasks.rados.rados.0.trial153.stdout:638: oid 5 version 63 is already newer than 63 2026-02-20T12:02:01.766 INFO:tasks.rados.rados.0.trial153.stdout:638: oid 5 version 63 is already newer than 63 2026-02-20T12:02:01.766 INFO:tasks.rados.rados.0.trial153.stdout:638: oid 5 version 63 is already newer than 60 2026-02-20T12:02:01.766 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 5 v 63 (ObjNum 312 snap 77 seq_num 312) dirty exists 2026-02-20T12:02:01.766 INFO:tasks.rados.rados.0.trial153.stdout:638: left oid 5 (ObjNum 312 snap 77 seq_num 312) 2026-02-20T12:02:02.584 INFO:tasks.rados.rados.0.trial153.stdout:633: done (6 left) 2026-02-20T12:02:02.584 INFO:tasks.rados.rados.0.trial153.stdout:634: done (5 left) 2026-02-20T12:02:02.584 INFO:tasks.rados.rados.0.trial153.stdout:635: done (4 left) 2026-02-20T12:02:02.584 INFO:tasks.rados.rados.0.trial153.stdout:636: done (3 left) 2026-02-20T12:02:02.584 INFO:tasks.rados.rados.0.trial153.stdout:637: done (2 left) 2026-02-20T12:02:02.585 INFO:tasks.rados.rados.0.trial153.stdout:638: done (1 left) 2026-02-20T12:02:02.585 INFO:tasks.rados.rados.0.trial153.stdout:639: done (0 left) 2026-02-20T12:02:02.585 INFO:tasks.rados.rados.0.trial153.stdout:640: write oid 3 current snap is 78 2026-02-20T12:02:02.585 INFO:tasks.rados.rados.0.trial153.stdout:640: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2026-02-20T12:02:02.589 INFO:tasks.rados.rados.0.trial153.stdout:640: writing trial15342161-3 from 460591 to 1184001 tid 1 2026-02-20T12:02:02.593 INFO:tasks.rados.rados.0.trial153.stdout:640: writing trial15342161-3 from 1716158 to 2119591 tid 2 2026-02-20T12:02:02.598 INFO:tasks.rados.rados.0.trial153.stdout:640: writing trial15342161-3 from 2729279 to 3169419 tid 3 2026-02-20T12:02:02.598 INFO:tasks.rados.rados.0.trial153.stdout:641: read oid 50 snap 76 2026-02-20T12:02:02.598 INFO:tasks.rados.rados.0.trial153.stdout:641: expect (ObjNum 265 snap 69 seq_num 265) 2026-02-20T12:02:02.598 INFO:tasks.rados.rados.0.trial153.stdout:642: delete oid 17 current snap is 78 2026-02-20T12:02:02.611 INFO:tasks.rados.rados.0.trial153.stdout:642: done (2 left) 2026-02-20T12:02:02.611 INFO:tasks.rados.rados.0.trial153.stdout:643: read oid 49 snap -1 2026-02-20T12:02:02.611 INFO:tasks.rados.rados.0.trial153.stdout:643: expect (ObjNum 302 snap 75 seq_num 302) 2026-02-20T12:02:02.611 INFO:tasks.rados.rados.0.trial153.stdout:644: snap_remove snap 77 2026-02-20T12:02:02.639 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:02.714 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:02.709+0000 7f0ea2d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:02.717 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:02.713+0000 7f0ea2d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:02.744 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:02.759 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:03.590 INFO:tasks.rados.rados.0.trial153.stdout:644: done (3 left) 2026-02-20T12:02:03.590 INFO:tasks.rados.rados.0.trial153.stdout:645: write oid 46 current snap is 78 2026-02-20T12:02:03.590 INFO:tasks.rados.rados.0.trial153.stdout:645: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2026-02-20T12:02:03.593 INFO:tasks.rados.rados.0.trial153.stdout:645: writing trial15342161-46 from 494205 to 968113 tid 1 2026-02-20T12:02:03.597 INFO:tasks.rados.rados.0.trial153.stdout:645: writing trial15342161-46 from 1433122 to 1840224 tid 2 2026-02-20T12:02:03.602 INFO:tasks.rados.rados.0.trial153.stdout:645: writing trial15342161-46 from 2384125 to 2846809 tid 3 2026-02-20T12:02:03.602 INFO:tasks.rados.rados.0.trial153.stdout:646: read oid 41 snap -1 2026-02-20T12:02:03.602 INFO:tasks.rados.rados.0.trial153.stdout:646: expect (ObjNum 274 snap 70 seq_num 274) 2026-02-20T12:02:03.602 INFO:tasks.rados.rados.0.trial153.stdout:647: read oid 26 snap -1 2026-02-20T12:02:03.602 INFO:tasks.rados.rados.0.trial153.stdout:647: expect (ObjNum 282 snap 70 seq_num 282) 2026-02-20T12:02:03.602 INFO:tasks.rados.rados.0.trial153.stdout:648: delete oid 4 current snap is 78 2026-02-20T12:02:03.602 INFO:tasks.rados.rados.0.trial153.stdout:641: expect (ObjNum 265 snap 69 seq_num 265) 2026-02-20T12:02:03.652 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:03.723 INFO:tasks.rados.rados.0.trial153.stdout:640: finishing write tid 1 to trial15342161-3 2026-02-20T12:02:03.723 INFO:tasks.rados.rados.0.trial153.stdout:643: expect (ObjNum 302 snap 75 seq_num 302) 2026-02-20T12:02:03.754 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2947272 2026-02-20T12:02:03.859 INFO:tasks.rados.rados.0.trial153.stdout:640: finishing write tid 2 to trial15342161-3 2026-02-20T12:02:03.859 INFO:tasks.rados.rados.0.trial153.stdout:641: done (6 left) 2026-02-20T12:02:03.859 INFO:tasks.rados.rados.0.trial153.stdout:643: done (5 left) 2026-02-20T12:02:03.859 INFO:tasks.rados.rados.0.trial153.stdout:648: done (4 left) 2026-02-20T12:02:03.859 INFO:tasks.rados.rados.0.trial153.stdout:649: read oid 47 snap -1 2026-02-20T12:02:03.859 INFO:tasks.rados.rados.0.trial153.stdout:649: expect (ObjNum 289 snap 72 seq_num 289) 2026-02-20T12:02:03.859 INFO:tasks.rados.rados.0.trial153.stdout:650: read oid 36 snap -1 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:650: expect deleted 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:651: read oid 30 snap -1 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:651: expect (ObjNum 227 snap 59 seq_num 227) 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:652: write oid 45 current snap is 78 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:640: finishing write tid 3 to trial15342161-3 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:640: finishing write tid 4 to trial15342161-3 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:640: finishing write tid 5 to trial15342161-3 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:640: oid 3 updating version 0 to 104 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:640: oid 3 version 104 is already newer than 104 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:640: oid 3 version 104 is already newer than 104 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:640: oid 3 version 104 is already newer than 104 2026-02-20T12:02:03.860 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 3 v 104 (ObjNum 313 snap 78 seq_num 313) dirty exists 2026-02-20T12:02:03.861 INFO:tasks.rados.rados.0.trial153.stdout:640: left oid 3 (ObjNum 313 snap 78 seq_num 313) 2026-02-20T12:02:03.861 INFO:tasks.rados.rados.0.trial153.stdout:652: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2026-02-20T12:02:03.862 INFO:tasks.rados.rados.0.trial153.stdout:652: writing trial15342161-45 from 527819 to 1152225 tid 1 2026-02-20T12:02:03.867 INFO:tasks.rados.rados.0.trial153.stdout:652: writing trial15342161-45 from 1833733 to 2244504 tid 2 2026-02-20T12:02:03.872 INFO:tasks.rados.rados.0.trial153.stdout:652: writing trial15342161-45 from 2722618 to 3324199 tid 3 2026-02-20T12:02:03.872 INFO:tasks.rados.rados.0.trial153.stdout:640: done (7 left) 2026-02-20T12:02:03.872 INFO:tasks.rados.rados.0.trial153.stdout:653: read oid 44 snap -1 2026-02-20T12:02:03.872 INFO:tasks.rados.rados.0.trial153.stdout:653: expect deleted 2026-02-20T12:02:03.872 INFO:tasks.rados.rados.0.trial153.stdout:654: write (excl) oid 35 current snap is 78 2026-02-20T12:02:03.872 INFO:tasks.rados.rados.0.trial153.stdout:654: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2026-02-20T12:02:03.877 INFO:tasks.rados.rados.0.trial153.stdout:654: writing trial15342161-35 from 544626 to 1244281 tid 1 2026-02-20T12:02:03.882 INFO:tasks.rados.rados.0.trial153.stdout:654: writing trial15342161-35 from 1892215 to 2446644 tid 2 2026-02-20T12:02:03.886 INFO:tasks.rados.rados.0.trial153.stdout:654: writing trial15342161-35 from 3150041 to 3344626 tid 3 2026-02-20T12:02:03.886 INFO:tasks.rados.rados.0.trial153.stdout:655: write (excl) oid 22 current snap is 78 2026-02-20T12:02:03.886 INFO:tasks.rados.rados.0.trial153.stdout:655: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2026-02-20T12:02:03.891 INFO:tasks.rados.rados.0.trial153.stdout:655: writing trial15342161-22 from 561433 to 1336337 tid 1 2026-02-20T12:02:03.895 INFO:tasks.rados.rados.0.trial153.stdout:655: writing trial15342161-22 from 1950697 to 2365137 tid 2 2026-02-20T12:02:03.899 INFO:tasks.rados.rados.0.trial153.stdout:655: writing trial15342161-22 from 2777464 to 3361433 tid 3 2026-02-20T12:02:03.899 INFO:tasks.rados.rados.0.trial153.stdout:656: write oid 29 current snap is 78 2026-02-20T12:02:03.899 INFO:tasks.rados.rados.0.trial153.stdout:656: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2026-02-20T12:02:03.903 INFO:tasks.rados.rados.0.trial153.stdout:656: writing trial15342161-29 from 578240 to 1144746 tid 1 2026-02-20T12:02:03.907 INFO:tasks.rados.rados.0.trial153.stdout:656: writing trial15342161-29 from 1609179 to 2167277 tid 2 2026-02-20T12:02:03.913 INFO:tasks.rados.rados.0.trial153.stdout:656: writing trial15342161-29 from 2804887 to 3356637 tid 3 2026-02-20T12:02:03.913 INFO:tasks.rados.rados.0.trial153.stdout:657: read oid 48 snap -1 2026-02-20T12:02:03.913 INFO:tasks.rados.rados.0.trial153.stdout:657: expect (ObjNum 308 snap 75 seq_num 308) 2026-02-20T12:02:03.913 INFO:tasks.rados.rados.0.trial153.stdout:658: delete oid 31 current snap is 78 2026-02-20T12:02:03.914 INFO:tasks.rados.rados.0.trial153.stdout:646: expect (ObjNum 274 snap 70 seq_num 274) 2026-02-20T12:02:03.914 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:03.950 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:03.972 INFO:tasks.rados.rados.0.trial153.stdout:647: expect (ObjNum 282 snap 70 seq_num 282) 2026-02-20T12:02:04.053 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2322023 2026-02-20T12:02:04.060 INFO:tasks.rados.rados.0.trial153.stdout:645: finishing write tid 1 to trial15342161-46 2026-02-20T12:02:04.060 INFO:tasks.rados.rados.0.trial153.stdout:646: done (12 left) 2026-02-20T12:02:04.060 INFO:tasks.rados.rados.0.trial153.stdout:647: done (11 left) 2026-02-20T12:02:04.060 INFO:tasks.rados.rados.0.trial153.stdout:658: done (10 left) 2026-02-20T12:02:04.061 INFO:tasks.rados.rados.0.trial153.stdout:659: snap_remove snap 76 2026-02-20T12:02:04.434 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~171s 2026-02-20T12:02:04.590 INFO:tasks.rados.rados.0.trial153.stdout:659: done (10 left) 2026-02-20T12:02:04.590 INFO:tasks.rados.rados.0.trial153.stdout:660: write (excl) oid 31 current snap is 78 2026-02-20T12:02:04.590 INFO:tasks.rados.rados.0.trial153.stdout:645: finishing write tid 2 to trial15342161-46 2026-02-20T12:02:04.590 INFO:tasks.rados.rados.0.trial153.stdout:660: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2026-02-20T12:02:04.594 INFO:tasks.rados.rados.0.trial153.stdout:660: writing trial15342161-31 from 611854 to 1328858 tid 1 2026-02-20T12:02:04.600 INFO:tasks.rados.rados.0.trial153.stdout:660: writing trial15342161-31 from 2126143 to 2687910 tid 2 2026-02-20T12:02:04.603 INFO:tasks.rados.rados.0.trial153.stdout:660: writing trial15342161-31 from 3259733 to 3411854 tid 3 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:661: snap_create 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:645: finishing write tid 3 to trial15342161-46 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:645: finishing write tid 4 to trial15342161-46 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:645: finishing write tid 5 to trial15342161-46 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:645: oid 46 updating version 0 to 33 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:645: oid 46 version 33 is already newer than 33 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:645: oid 46 version 33 is already newer than 33 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:645: oid 46 version 33 is already newer than 33 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 46 v 33 (ObjNum 315 snap 78 seq_num 315) dirty exists 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:645: left oid 46 (ObjNum 315 snap 78 seq_num 315) 2026-02-20T12:02:04.604 INFO:tasks.rados.rados.0.trial153.stdout:649: expect (ObjNum 289 snap 72 seq_num 289) 2026-02-20T12:02:04.605 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 661457 2026-02-20T12:02:04.655 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:04.687 INFO:tasks.rados.rados.0.trial153.stdout:652: finishing write tid 1 to trial15342161-45 2026-02-20T12:02:04.687 INFO:tasks.rados.rados.0.trial153.stdout:651: expect (ObjNum 227 snap 59 seq_num 227) 2026-02-20T12:02:04.688 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:04.720 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:04.753 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 133790 2026-02-20T12:02:04.760 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:04.785 INFO:tasks.rados.rados.0.trial153.stdout:652: finishing write tid 2 to trial15342161-45 2026-02-20T12:02:04.785 INFO:tasks.rados.rados.0.trial153.stdout:652: finishing write tid 3 to trial15342161-45 2026-02-20T12:02:04.785 INFO:tasks.rados.rados.0.trial153.stdout:652: finishing write tid 4 to trial15342161-45 2026-02-20T12:02:04.785 INFO:tasks.rados.rados.0.trial153.stdout:652: finishing write tid 5 to trial15342161-45 2026-02-20T12:02:04.785 INFO:tasks.rados.rados.0.trial153.stdout:652: oid 45 updating version 0 to 61 2026-02-20T12:02:04.786 INFO:tasks.rados.rados.0.trial153.stdout:652: oid 45 version 61 is already newer than 61 2026-02-20T12:02:04.786 INFO:tasks.rados.rados.0.trial153.stdout:652: oid 45 version 61 is already newer than 61 2026-02-20T12:02:04.786 INFO:tasks.rados.rados.0.trial153.stdout:652: oid 45 version 61 is already newer than 61 2026-02-20T12:02:04.786 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 61 (ObjNum 317 snap 78 seq_num 317) dirty exists 2026-02-20T12:02:04.786 INFO:tasks.rados.rados.0.trial153.stdout:652: left oid 45 (ObjNum 317 snap 78 seq_num 317) 2026-02-20T12:02:04.786 INFO:tasks.rados.rados.0.trial153.stdout:654: finishing write tid 1 to trial15342161-35 2026-02-20T12:02:04.786 INFO:tasks.rados.rados.0.trial153.stdout:657: expect (ObjNum 308 snap 75 seq_num 308) 2026-02-20T12:02:04.820 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2268748 2026-02-20T12:02:04.835 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:04.829+0000 7fb6524cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:04.837 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:04.832+0000 7fb6524cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:04.864 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:04.870 INFO:tasks.rados.rados.0.trial153.stdout:656: finishing write tid 1 to trial15342161-29 2026-02-20T12:02:04.871 INFO:tasks.rados.rados.0.trial153.stdout:654: finishing write tid 2 to trial15342161-35 2026-02-20T12:02:04.871 INFO:tasks.rados.rados.0.trial153.stdout:656: finishing write tid 2 to trial15342161-29 2026-02-20T12:02:04.871 INFO:tasks.rados.rados.0.trial153.stdout:654: finishing write tid 3 to trial15342161-35 2026-02-20T12:02:04.871 INFO:tasks.rados.rados.0.trial153.stdout:654: finishing write tid 4 to trial15342161-35 2026-02-20T12:02:04.871 INFO:tasks.rados.rados.0.trial153.stdout:656: finishing write tid 3 to trial15342161-29 2026-02-20T12:02:04.871 INFO:tasks.rados.rados.0.trial153.stdout:654: finishing write tid 5 to trial15342161-35 2026-02-20T12:02:04.871 INFO:tasks.rados.rados.0.trial153.stdout:654: oid 35 updating version 0 to 114 2026-02-20T12:02:04.871 INFO:tasks.rados.rados.0.trial153.stdout:654: oid 35 version 114 is already newer than 114 2026-02-20T12:02:04.871 INFO:tasks.rados.rados.0.trial153.stdout:654: oid 35 version 114 is already newer than 111 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:654: oid 35 version 114 is already newer than 114 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 35 v 114 (ObjNum 318 snap 78 seq_num 318) dirty exists 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:654: left oid 35 (ObjNum 318 snap 78 seq_num 318) 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:656: finishing write tid 4 to trial15342161-29 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:655: finishing write tid 1 to trial15342161-22 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:656: finishing write tid 5 to trial15342161-29 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:656: oid 29 updating version 0 to 217 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:656: oid 29 version 217 is already newer than 217 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:656: oid 29 version 217 is already newer than 217 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:656: oid 29 version 217 is already newer than 217 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 29 v 217 (ObjNum 320 snap 78 seq_num 320) dirty exists 2026-02-20T12:02:04.872 INFO:tasks.rados.rados.0.trial153.stdout:656: left oid 29 (ObjNum 320 snap 78 seq_num 320) 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:655: finishing write tid 2 to trial15342161-22 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:655: finishing write tid 3 to trial15342161-22 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:655: finishing write tid 4 to trial15342161-22 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:655: finishing write tid 5 to trial15342161-22 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:655: oid 22 updating version 0 to 118 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:655: oid 22 version 118 is already newer than 118 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:655: oid 22 version 118 is already newer than 115 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:655: oid 22 version 118 is already newer than 118 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 22 v 118 (ObjNum 319 snap 78 seq_num 319) dirty exists 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:655: left oid 22 (ObjNum 319 snap 78 seq_num 319) 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:660: finishing write tid 1 to trial15342161-31 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:660: finishing write tid 2 to trial15342161-31 2026-02-20T12:02:04.873 INFO:tasks.rados.rados.0.trial153.stdout:660: finishing write tid 3 to trial15342161-31 2026-02-20T12:02:04.874 INFO:tasks.rados.rados.0.trial153.stdout:660: finishing write tid 4 to trial15342161-31 2026-02-20T12:02:04.874 INFO:tasks.rados.rados.0.trial153.stdout:660: finishing write tid 5 to trial15342161-31 2026-02-20T12:02:04.874 INFO:tasks.rados.rados.0.trial153.stdout:660: oid 31 updating version 0 to 54 2026-02-20T12:02:04.874 INFO:tasks.rados.rados.0.trial153.stdout:660: oid 31 updating version 54 to 57 2026-02-20T12:02:04.874 INFO:tasks.rados.rados.0.trial153.stdout:660: oid 31 version 57 is already newer than 57 2026-02-20T12:02:04.874 INFO:tasks.rados.rados.0.trial153.stdout:660: oid 31 version 57 is already newer than 57 2026-02-20T12:02:04.874 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 31 v 57 (ObjNum 322 snap 78 seq_num 322) dirty exists 2026-02-20T12:02:04.874 INFO:tasks.rados.rados.0.trial153.stdout:660: left oid 31 (ObjNum 322 snap 78 seq_num 322) 2026-02-20T12:02:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:04.881 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:04.898 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:04.968 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:04.970 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:04.971 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:05.594 INFO:tasks.rados.rados.0.trial153.stdout:645: done (11 left) 2026-02-20T12:02:05.594 INFO:tasks.rados.rados.0.trial153.stdout:649: done (10 left) 2026-02-20T12:02:05.595 INFO:tasks.rados.rados.0.trial153.stdout:650: done (9 left) 2026-02-20T12:02:05.595 INFO:tasks.rados.rados.0.trial153.stdout:651: done (8 left) 2026-02-20T12:02:05.595 INFO:tasks.rados.rados.0.trial153.stdout:652: done (7 left) 2026-02-20T12:02:05.595 INFO:tasks.rados.rados.0.trial153.stdout:653: done (6 left) 2026-02-20T12:02:05.595 INFO:tasks.rados.rados.0.trial153.stdout:654: done (5 left) 2026-02-20T12:02:05.595 INFO:tasks.rados.rados.0.trial153.stdout:655: done (4 left) 2026-02-20T12:02:05.595 INFO:tasks.rados.rados.0.trial153.stdout:656: done (3 left) 2026-02-20T12:02:05.595 INFO:tasks.rados.rados.0.trial153.stdout:657: done (2 left) 2026-02-20T12:02:05.595 INFO:tasks.rados.rados.0.trial153.stdout:660: done (1 left) 2026-02-20T12:02:05.596 INFO:tasks.rados.rados.0.trial153.stdout:661: done (0 left) 2026-02-20T12:02:05.596 INFO:tasks.rados.rados.0.trial153.stdout:662: snap_remove snap 78 2026-02-20T12:02:06.516 INFO:tasks.rados.rados.0.trial153.stdout:662: done (0 left) 2026-02-20T12:02:06.516 INFO:tasks.rados.rados.0.trial153.stdout:663: read oid 45 snap -1 2026-02-20T12:02:06.516 INFO:tasks.rados.rados.0.trial153.stdout:663: expect (ObjNum 317 snap 78 seq_num 317) 2026-02-20T12:02:06.516 INFO:tasks.rados.rados.0.trial153.stdout:664: delete oid 28 current snap is 79 2026-02-20T12:02:06.546 INFO:tasks.rados.rados.0.trial153.stdout:663: expect (ObjNum 317 snap 78 seq_num 317) 2026-02-20T12:02:06.546 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 308473 2026-02-20T12:02:06.609 INFO:tasks.rados.rados.0.trial153.stdout:663: done (1 left) 2026-02-20T12:02:06.610 INFO:tasks.rados.rados.0.trial153.stdout:664: done (0 left) 2026-02-20T12:02:06.610 INFO:tasks.rados.rados.0.trial153.stdout:665: read oid 19 snap -1 2026-02-20T12:02:06.610 INFO:tasks.rados.rados.0.trial153.stdout:665: expect (ObjNum 259 snap 68 seq_num 259) 2026-02-20T12:02:06.610 INFO:tasks.rados.rados.0.trial153.stdout:666: read oid 22 snap -1 2026-02-20T12:02:06.610 INFO:tasks.rados.rados.0.trial153.stdout:666: expect (ObjNum 319 snap 78 seq_num 319) 2026-02-20T12:02:06.610 INFO:tasks.rados.rados.0.trial153.stdout:667: snap_create 2026-02-20T12:02:06.623 INFO:tasks.rados.rados.0.trial153.stdout:666: expect (ObjNum 319 snap 78 seq_num 319) 2026-02-20T12:02:06.637 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:06.692 INFO:tasks.rados.rados.0.trial153.stdout:665: expect (ObjNum 259 snap 68 seq_num 259) 2026-02-20T12:02:06.725 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2176326 2026-02-20T12:02:06.883 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:06.957 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:06.952+0000 7fb5a710b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:06.960 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:06.956+0000 7fb5a710b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:06.987 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:07.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:07.599 INFO:tasks.rados.rados.0.trial153.stdout:665: done (2 left) 2026-02-20T12:02:07.599 INFO:tasks.rados.rados.0.trial153.stdout:666: done (1 left) 2026-02-20T12:02:07.599 INFO:tasks.rados.rados.0.trial153.stdout:667: done (0 left) 2026-02-20T12:02:07.599 INFO:tasks.rados.rados.0.trial153.stdout:668: delete oid 13 current snap is 80 2026-02-20T12:02:07.639 INFO:tasks.rados.rados.0.trial153.stdout:668: done (0 left) 2026-02-20T12:02:07.640 INFO:tasks.rados.rados.0.trial153.stdout:669: delete oid 45 current snap is 80 2026-02-20T12:02:07.664 INFO:tasks.rados.rados.0.trial153.stdout:669: done (0 left) 2026-02-20T12:02:07.664 INFO:tasks.rados.rados.0.trial153.stdout:670: delete oid 48 current snap is 80 2026-02-20T12:02:07.691 INFO:tasks.rados.rados.0.trial153.stdout:670: done (0 left) 2026-02-20T12:02:07.692 INFO:tasks.rados.rados.0.trial153.stdout:671: read oid 11 snap -1 2026-02-20T12:02:07.692 INFO:tasks.rados.rados.0.trial153.stdout:671: expect deleted 2026-02-20T12:02:07.692 INFO:tasks.rados.rados.0.trial153.stdout:672: write (excl) oid 44 current snap is 80 2026-02-20T12:02:07.692 INFO:tasks.rados.rados.0.trial153.stdout:672: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2026-02-20T12:02:07.695 INFO:tasks.rados.rados.0.trial153.stdout:672: writing trial15342161-44 from 695889 to 1105491 tid 1 2026-02-20T12:02:07.700 INFO:tasks.rados.rados.0.trial153.stdout:672: writing trial15342161-44 from 1618553 to 2331316 tid 2 2026-02-20T12:02:07.705 INFO:tasks.rados.rados.0.trial153.stdout:672: writing trial15342161-44 from 2996848 to 3495889 tid 3 2026-02-20T12:02:07.705 INFO:tasks.rados.rados.0.trial153.stdout:673: read oid 17 snap -1 2026-02-20T12:02:07.705 INFO:tasks.rados.rados.0.trial153.stdout:673: expect deleted 2026-02-20T12:02:07.705 INFO:tasks.rados.rados.0.trial153.stdout:674: delete oid 29 current snap is 80 2026-02-20T12:02:07.712 INFO:tasks.rados.rados.0.trial153.stdout:672: finishing write tid 1 to trial15342161-44 2026-02-20T12:02:07.729 INFO:tasks.rados.rados.0.trial153.stdout:672: finishing write tid 2 to trial15342161-44 2026-02-20T12:02:07.731 INFO:tasks.rados.rados.0.trial153.stdout:671: done (3 left) 2026-02-20T12:02:07.731 INFO:tasks.rados.rados.0.trial153.stdout:673: done (2 left) 2026-02-20T12:02:07.731 INFO:tasks.rados.rados.0.trial153.stdout:674: done (1 left) 2026-02-20T12:02:07.731 INFO:tasks.rados.rados.0.trial153.stdout:675: write oid 2 current snap is 80 2026-02-20T12:02:07.732 INFO:tasks.rados.rados.0.trial153.stdout:675: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2026-02-20T12:02:07.736 INFO:tasks.rados.rados.0.trial153.stdout:675: writing trial15342161-2 from 729503 to 1289603 tid 1 2026-02-20T12:02:07.742 INFO:tasks.rados.rados.0.trial153.stdout:675: writing trial15342161-2 from 2019164 to 2735596 tid 2 2026-02-20T12:02:07.745 INFO:tasks.rados.rados.0.trial153.stdout:675: writing trial15342161-2 from 3335341 to 3529503 tid 3 2026-02-20T12:02:07.745 INFO:tasks.rados.rados.0.trial153.stdout:676: write oid 12 current snap is 80 2026-02-20T12:02:07.745 INFO:tasks.rados.rados.0.trial153.stdout:672: finishing write tid 3 to trial15342161-44 2026-02-20T12:02:07.745 INFO:tasks.rados.rados.0.trial153.stdout:676: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2026-02-20T12:02:07.750 INFO:tasks.rados.rados.0.trial153.stdout:676: writing trial15342161-12 from 746310 to 1381659 tid 1 2026-02-20T12:02:07.755 INFO:tasks.rados.rados.0.trial153.stdout:676: writing trial15342161-12 from 2077646 to 2654089 tid 2 2026-02-20T12:02:07.758 INFO:tasks.rados.rados.0.trial153.stdout:676: writing trial15342161-12 from 3362764 to 3546310 tid 3 2026-02-20T12:02:07.758 INFO:tasks.rados.rados.0.trial153.stdout:677: write (excl) oid 1 current snap is 80 2026-02-20T12:02:07.758 INFO:tasks.rados.rados.0.trial153.stdout:672: finishing write tid 4 to trial15342161-44 2026-02-20T12:02:07.759 INFO:tasks.rados.rados.0.trial153.stdout:677: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2026-02-20T12:02:07.763 INFO:tasks.rados.rados.0.trial153.stdout:677: writing trial15342161-1 from 763117 to 1473715 tid 1 2026-02-20T12:02:07.769 INFO:tasks.rados.rados.0.trial153.stdout:677: writing trial15342161-1 from 2136128 to 2856229 tid 2 2026-02-20T12:02:07.771 INFO:tasks.rados.rados.0.trial153.stdout:677: writing trial15342161-1 from 3390187 to 3563117 tid 3 2026-02-20T12:02:07.772 INFO:tasks.rados.rados.0.trial153.stdout:678: snap_remove snap 79 2026-02-20T12:02:08.601 INFO:tasks.rados.rados.0.trial153.stdout:678: done (4 left) 2026-02-20T12:02:08.602 INFO:tasks.rados.rados.0.trial153.stdout:679: read oid 45 snap -1 2026-02-20T12:02:08.602 INFO:tasks.rados.rados.0.trial153.stdout:679: expect deleted 2026-02-20T12:02:08.602 INFO:tasks.rados.rados.0.trial153.stdout:680: write oid 26 current snap is 80 2026-02-20T12:02:08.602 INFO:tasks.rados.rados.0.trial153.stdout:672: finishing write tid 5 to trial15342161-44 2026-02-20T12:02:08.602 INFO:tasks.rados.rados.0.trial153.stdout:672: oid 44 updating version 0 to 113 2026-02-20T12:02:08.602 INFO:tasks.rados.rados.0.trial153.stdout:672: oid 44 version 113 is already newer than 113 2026-02-20T12:02:08.602 INFO:tasks.rados.rados.0.trial153.stdout:672: oid 44 version 113 is already newer than 113 2026-02-20T12:02:08.602 INFO:tasks.rados.rados.0.trial153.stdout:672: oid 44 version 113 is already newer than 110 2026-02-20T12:02:08.602 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 44 v 113 (ObjNum 327 snap 80 seq_num 327) dirty exists 2026-02-20T12:02:08.603 INFO:tasks.rados.rados.0.trial153.stdout:672: left oid 44 (ObjNum 327 snap 80 seq_num 327) 2026-02-20T12:02:08.603 INFO:tasks.rados.rados.0.trial153.stdout:680: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2026-02-20T12:02:08.607 INFO:tasks.rados.rados.0.trial153.stdout:680: writing trial15342161-26 from 779924 to 1565771 tid 1 2026-02-20T12:02:08.612 INFO:tasks.rados.rados.0.trial153.stdout:680: writing trial15342161-26 from 2078257 to 2658369 tid 2 2026-02-20T12:02:08.616 INFO:tasks.rados.rados.0.trial153.stdout:680: writing trial15342161-26 from 3301257 to 3579924 tid 3 2026-02-20T12:02:08.617 INFO:tasks.rados.rados.0.trial153.stdout:672: done (5 left) 2026-02-20T12:02:08.617 INFO:tasks.rados.rados.0.trial153.stdout:681: write (excl) oid 20 current snap is 80 2026-02-20T12:02:08.617 INFO:tasks.rados.rados.0.trial153.stdout:675: finishing write tid 1 to trial15342161-2 2026-02-20T12:02:08.617 INFO:tasks.rados.rados.0.trial153.stdout:681: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2026-02-20T12:02:08.621 INFO:tasks.rados.rados.0.trial153.stdout:681: writing trial15342161-20 from 796731 to 1257827 tid 1 2026-02-20T12:02:08.626 INFO:tasks.rados.rados.0.trial153.stdout:681: writing trial15342161-20 from 1736739 to 2460509 tid 2 2026-02-20T12:02:08.632 INFO:tasks.rados.rados.0.trial153.stdout:681: writing trial15342161-20 from 2928680 to 3596731 tid 3 2026-02-20T12:02:08.632 INFO:tasks.rados.rados.0.trial153.stdout:682: delete oid 7 current snap is 80 2026-02-20T12:02:08.633 INFO:tasks.rados.rados.0.trial153.stdout:675: finishing write tid 2 to trial15342161-2 2026-02-20T12:02:08.633 INFO:tasks.rados.rados.0.trial153.stdout:676: finishing write tid 1 to trial15342161-12 2026-02-20T12:02:08.633 INFO:tasks.rados.rados.0.trial153.stdout:675: finishing write tid 3 to trial15342161-2 2026-02-20T12:02:08.633 INFO:tasks.rados.rados.0.trial153.stdout:676: finishing write tid 2 to trial15342161-12 2026-02-20T12:02:08.633 INFO:tasks.rados.rados.0.trial153.stdout:675: finishing write tid 4 to trial15342161-2 2026-02-20T12:02:08.633 INFO:tasks.rados.rados.0.trial153.stdout:675: finishing write tid 5 to trial15342161-2 2026-02-20T12:02:08.633 INFO:tasks.rados.rados.0.trial153.stdout:675: oid 2 updating version 0 to 228 2026-02-20T12:02:08.633 INFO:tasks.rados.rados.0.trial153.stdout:675: oid 2 version 228 is already newer than 228 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:675: oid 2 version 228 is already newer than 228 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:675: oid 2 version 228 is already newer than 228 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 2 v 228 (ObjNum 329 snap 80 seq_num 329) dirty exists 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:675: left oid 2 (ObjNum 329 snap 80 seq_num 329) 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:677: finishing write tid 1 to trial15342161-1 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:676: finishing write tid 3 to trial15342161-12 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:676: finishing write tid 4 to trial15342161-12 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:676: finishing write tid 5 to trial15342161-12 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:676: oid 12 updating version 0 to 38 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:676: oid 12 version 38 is already newer than 38 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:676: oid 12 version 38 is already newer than 38 2026-02-20T12:02:08.634 INFO:tasks.rados.rados.0.trial153.stdout:676: oid 12 version 38 is already newer than 38 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 12 v 38 (ObjNum 330 snap 80 seq_num 330) dirty exists 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:676: left oid 12 (ObjNum 330 snap 80 seq_num 330) 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:677: finishing write tid 2 to trial15342161-1 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:677: finishing write tid 3 to trial15342161-1 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:677: finishing write tid 4 to trial15342161-1 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:677: finishing write tid 5 to trial15342161-1 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:677: oid 1 updating version 0 to 229 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:677: oid 1 updating version 229 to 232 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:677: oid 1 version 232 is already newer than 232 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:677: oid 1 version 232 is already newer than 232 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 1 v 232 (ObjNum 331 snap 80 seq_num 331) dirty exists 2026-02-20T12:02:08.635 INFO:tasks.rados.rados.0.trial153.stdout:677: left oid 1 (ObjNum 331 snap 80 seq_num 331) 2026-02-20T12:02:08.641 INFO:tasks.rados.rados.0.trial153.stdout:681: finishing write tid 1 to trial15342161-20 2026-02-20T12:02:08.647 INFO:tasks.rados.rados.0.trial153.stdout:680: finishing write tid 1 to trial15342161-26 2026-02-20T12:02:08.656 INFO:tasks.rados.rados.0.trial153.stdout:680: finishing write tid 2 to trial15342161-26 2026-02-20T12:02:08.666 INFO:tasks.rados.rados.0.trial153.stdout:681: finishing write tid 2 to trial15342161-20 2026-02-20T12:02:08.669 INFO:tasks.rados.rados.0.trial153.stdout:680: finishing write tid 3 to trial15342161-26 2026-02-20T12:02:08.670 INFO:tasks.rados.rados.0.trial153.stdout:675: done (6 left) 2026-02-20T12:02:08.670 INFO:tasks.rados.rados.0.trial153.stdout:676: done (5 left) 2026-02-20T12:02:08.670 INFO:tasks.rados.rados.0.trial153.stdout:677: done (4 left) 2026-02-20T12:02:08.670 INFO:tasks.rados.rados.0.trial153.stdout:679: done (3 left) 2026-02-20T12:02:08.670 INFO:tasks.rados.rados.0.trial153.stdout:682: done (2 left) 2026-02-20T12:02:08.670 INFO:tasks.rados.rados.0.trial153.stdout:683: write (excl) oid 35 current snap is 80 2026-02-20T12:02:08.671 INFO:tasks.rados.rados.0.trial153.stdout:683: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2026-02-20T12:02:08.674 INFO:tasks.rados.rados.0.trial153.stdout:683: writing trial15342161-35 from 430345 to 1158292 tid 1 2026-02-20T12:02:08.679 INFO:tasks.rados.rados.0.trial153.stdout:683: writing trial15342161-35 from 1570056 to 2297495 tid 2 2026-02-20T12:02:08.683 INFO:tasks.rados.rados.0.trial153.stdout:683: writing trial15342161-35 from 2699879 to 3169768 tid 3 2026-02-20T12:02:08.683 INFO:tasks.rados.rados.0.trial153.stdout:684: read oid 16 snap -1 2026-02-20T12:02:08.683 INFO:tasks.rados.rados.0.trial153.stdout:684: expect (ObjNum 287 snap 72 seq_num 287) 2026-02-20T12:02:08.683 INFO:tasks.rados.rados.0.trial153.stdout:685: write oid 21 current snap is 80 2026-02-20T12:02:08.683 INFO:tasks.rados.rados.0.trial153.stdout:685: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2026-02-20T12:02:08.686 INFO:tasks.rados.rados.0.trial153.stdout:685: writing trial15342161-21 from 447152 to 850348 tid 1 2026-02-20T12:02:08.691 INFO:tasks.rados.rados.0.trial153.stdout:685: writing trial15342161-21 from 1512185 to 2099635 tid 2 2026-02-20T12:02:08.698 INFO:tasks.rados.rados.0.trial153.stdout:685: writing trial15342161-21 from 2727302 to 3408463 tid 3 2026-02-20T12:02:08.698 INFO:tasks.rados.rados.0.trial153.stdout:686: delete oid 23 current snap is 80 2026-02-20T12:02:08.698 INFO:tasks.rados.rados.0.trial153.stdout:681: finishing write tid 3 to trial15342161-20 2026-02-20T12:02:08.698 INFO:tasks.rados.rados.0.trial153.stdout:680: finishing write tid 4 to trial15342161-26 2026-02-20T12:02:08.698 INFO:tasks.rados.rados.0.trial153.stdout:681: finishing write tid 4 to trial15342161-20 2026-02-20T12:02:08.698 INFO:tasks.rados.rados.0.trial153.stdout:680: finishing write tid 5 to trial15342161-26 2026-02-20T12:02:08.698 INFO:tasks.rados.rados.0.trial153.stdout:680: oid 26 updating version 0 to 169 2026-02-20T12:02:08.698 INFO:tasks.rados.rados.0.trial153.stdout:680: oid 26 version 169 is already newer than 169 2026-02-20T12:02:08.698 INFO:tasks.rados.rados.0.trial153.stdout:680: oid 26 version 169 is already newer than 169 2026-02-20T12:02:08.698 INFO:tasks.rados.rados.0.trial153.stdout:680: oid 26 version 169 is already newer than 169 2026-02-20T12:02:08.699 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 26 v 169 (ObjNum 332 snap 80 seq_num 332) dirty exists 2026-02-20T12:02:08.699 INFO:tasks.rados.rados.0.trial153.stdout:680: left oid 26 (ObjNum 332 snap 80 seq_num 332) 2026-02-20T12:02:08.699 INFO:tasks.rados.rados.0.trial153.stdout:681: finishing write tid 5 to trial15342161-20 2026-02-20T12:02:08.699 INFO:tasks.rados.rados.0.trial153.stdout:681: oid 20 updating version 0 to 61 2026-02-20T12:02:08.699 INFO:tasks.rados.rados.0.trial153.stdout:681: oid 20 version 61 is already newer than 61 2026-02-20T12:02:08.699 INFO:tasks.rados.rados.0.trial153.stdout:681: oid 20 version 61 is already newer than 58 2026-02-20T12:02:08.699 INFO:tasks.rados.rados.0.trial153.stdout:681: oid 20 version 61 is already newer than 61 2026-02-20T12:02:08.699 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 20 v 61 (ObjNum 333 snap 80 seq_num 333) dirty exists 2026-02-20T12:02:08.699 INFO:tasks.rados.rados.0.trial153.stdout:681: left oid 20 (ObjNum 333 snap 80 seq_num 333) 2026-02-20T12:02:08.699 INFO:tasks.rados.rados.0.trial153.stdout:683: finishing write tid 1 to trial15342161-35 2026-02-20T12:02:08.716 INFO:tasks.rados.rados.0.trial153.stdout:683: finishing write tid 2 to trial15342161-35 2026-02-20T12:02:08.722 INFO:tasks.rados.rados.0.trial153.stdout:684: expect (ObjNum 287 snap 72 seq_num 287) 2026-02-20T12:02:08.739 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:08.787 INFO:tasks.rados.rados.0.trial153.stdout:683: finishing write tid 3 to trial15342161-35 2026-02-20T12:02:08.787 INFO:tasks.rados.rados.0.trial153.stdout:685: finishing write tid 1 to trial15342161-21 2026-02-20T12:02:08.787 INFO:tasks.rados.rados.0.trial153.stdout:680: done (5 left) 2026-02-20T12:02:08.787 INFO:tasks.rados.rados.0.trial153.stdout:681: done (4 left) 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:684: done (3 left) 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:686: done (2 left) 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:687: delete oid 48 current snap is 80 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:683: finishing write tid 4 to trial15342161-35 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:685: finishing write tid 2 to trial15342161-21 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:683: finishing write tid 5 to trial15342161-35 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:683: oid 35 updating version 0 to 123 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:683: oid 35 version 123 is already newer than 123 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:683: oid 35 version 123 is already newer than 120 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:683: oid 35 version 123 is already newer than 123 2026-02-20T12:02:08.788 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 35 v 123 (ObjNum 335 snap 80 seq_num 335) dirty exists 2026-02-20T12:02:08.789 INFO:tasks.rados.rados.0.trial153.stdout:683: left oid 35 (ObjNum 335 snap 80 seq_num 335) 2026-02-20T12:02:08.789 INFO:tasks.rados.rados.0.trial153.stdout:685: finishing write tid 3 to trial15342161-21 2026-02-20T12:02:08.789 INFO:tasks.rados.rados.0.trial153.stdout:685: finishing write tid 4 to trial15342161-21 2026-02-20T12:02:08.789 INFO:tasks.rados.rados.0.trial153.stdout:685: finishing write tid 5 to trial15342161-21 2026-02-20T12:02:08.789 INFO:tasks.rados.rados.0.trial153.stdout:685: oid 21 updating version 0 to 117 2026-02-20T12:02:08.789 INFO:tasks.rados.rados.0.trial153.stdout:685: oid 21 version 117 is already newer than 117 2026-02-20T12:02:08.789 INFO:tasks.rados.rados.0.trial153.stdout:685: oid 21 version 117 is already newer than 117 2026-02-20T12:02:08.789 INFO:tasks.rados.rados.0.trial153.stdout:685: oid 21 version 117 is already newer than 117 2026-02-20T12:02:08.789 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 21 v 117 (ObjNum 336 snap 80 seq_num 336) dirty exists 2026-02-20T12:02:08.789 INFO:tasks.rados.rados.0.trial153.stdout:685: left oid 21 (ObjNum 336 snap 80 seq_num 336) 2026-02-20T12:02:08.796 INFO:tasks.rados.rados.0.trial153.stdout:683: done (2 left) 2026-02-20T12:02:08.796 INFO:tasks.rados.rados.0.trial153.stdout:685: done (1 left) 2026-02-20T12:02:08.796 INFO:tasks.rados.rados.0.trial153.stdout:687: done (0 left) 2026-02-20T12:02:08.796 INFO:tasks.rados.rados.0.trial153.stdout:688: read oid 48 snap -1 2026-02-20T12:02:08.796 INFO:tasks.rados.rados.0.trial153.stdout:688: expect deleted 2026-02-20T12:02:08.797 INFO:tasks.rados.rados.0.trial153.stdout:689: write (excl) oid 20 current snap is 80 2026-02-20T12:02:08.797 INFO:tasks.rados.rados.0.trial153.stdout:689: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2026-02-20T12:02:08.800 INFO:tasks.rados.rados.0.trial153.stdout:689: writing trial15342161-20 from 497573 to 1126516 tid 1 2026-02-20T12:02:08.805 INFO:tasks.rados.rados.0.trial153.stdout:689: writing trial15342161-20 from 1687631 to 2422408 tid 2 2026-02-20T12:02:08.810 INFO:tasks.rados.rados.0.trial153.stdout:689: writing trial15342161-20 from 3093218 to 3697573 tid 3 2026-02-20T12:02:08.810 INFO:tasks.rados.rados.0.trial153.stdout:690: write (excl) oid 14 current snap is 80 2026-02-20T12:02:08.811 INFO:tasks.rados.rados.0.trial153.stdout:690: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2026-02-20T12:02:08.815 INFO:tasks.rados.rados.0.trial153.stdout:690: writing trial15342161-14 from 514380 to 1218572 tid 1 2026-02-20T12:02:08.818 INFO:tasks.rados.rados.0.trial153.stdout:690: writing trial15342161-14 from 1629760 to 2108195 tid 2 2026-02-20T12:02:08.822 INFO:tasks.rados.rados.0.trial153.stdout:690: writing trial15342161-14 from 2604288 to 3046890 tid 3 2026-02-20T12:02:08.822 INFO:tasks.rados.rados.0.trial153.stdout:691: snap_create 2026-02-20T12:02:08.822 INFO:tasks.rados.rados.0.trial153.stdout:689: finishing write tid 1 to trial15342161-20 2026-02-20T12:02:08.844 INFO:tasks.rados.rados.0.trial153.stdout:690: finishing write tid 1 to trial15342161-14 2026-02-20T12:02:08.861 INFO:tasks.rados.rados.0.trial153.stdout:689: finishing write tid 2 to trial15342161-20 2026-02-20T12:02:08.867 INFO:tasks.rados.rados.0.trial153.stdout:690: finishing write tid 2 to trial15342161-14 2026-02-20T12:02:08.878 INFO:tasks.rados.rados.0.trial153.stdout:689: finishing write tid 3 to trial15342161-20 2026-02-20T12:02:08.879 INFO:tasks.rados.rados.0.trial153.stdout:690: finishing write tid 3 to trial15342161-14 2026-02-20T12:02:08.888 INFO:tasks.rados.rados.0.trial153.stdout:689: finishing write tid 4 to trial15342161-20 2026-02-20T12:02:08.888 INFO:tasks.rados.rados.0.trial153.stdout:690: finishing write tid 4 to trial15342161-14 2026-02-20T12:02:08.890 INFO:tasks.rados.rados.0.trial153.stdout:690: finishing write tid 5 to trial15342161-14 2026-02-20T12:02:08.890 INFO:tasks.rados.rados.0.trial153.stdout:690: oid 14 updating version 0 to 179 2026-02-20T12:02:08.890 INFO:tasks.rados.rados.0.trial153.stdout:690: oid 14 updating version 179 to 182 2026-02-20T12:02:08.890 INFO:tasks.rados.rados.0.trial153.stdout:690: oid 14 version 182 is already newer than 182 2026-02-20T12:02:08.890 INFO:tasks.rados.rados.0.trial153.stdout:690: oid 14 version 182 is already newer than 182 2026-02-20T12:02:08.890 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 182 (ObjNum 340 snap 80 seq_num 340) dirty exists 2026-02-20T12:02:08.891 INFO:tasks.rados.rados.0.trial153.stdout:690: left oid 14 (ObjNum 340 snap 80 seq_num 340) 2026-02-20T12:02:08.891 INFO:tasks.rados.rados.0.trial153.stdout:689: finishing write tid 5 to trial15342161-20 2026-02-20T12:02:08.891 INFO:tasks.rados.rados.0.trial153.stdout:689: oid 20 updating version 0 to 65 2026-02-20T12:02:08.891 INFO:tasks.rados.rados.0.trial153.stdout:689: oid 20 version 65 is already newer than 65 2026-02-20T12:02:08.891 INFO:tasks.rados.rados.0.trial153.stdout:689: oid 20 version 65 is already newer than 65 2026-02-20T12:02:08.891 INFO:tasks.rados.rados.0.trial153.stdout:689: oid 20 version 65 is already newer than 62 2026-02-20T12:02:08.891 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 20 v 65 (ObjNum 339 snap 80 seq_num 339) dirty exists 2026-02-20T12:02:08.891 INFO:tasks.rados.rados.0.trial153.stdout:689: left oid 20 (ObjNum 339 snap 80 seq_num 339) 2026-02-20T12:02:09.003 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:09.080 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:09.074+0000 7f91d905a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:09.082 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:09.078+0000 7f91d905a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:09.108 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:09.125 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:09.603 INFO:tasks.rados.rados.0.trial153.stdout:688: done (3 left) 2026-02-20T12:02:09.603 INFO:tasks.rados.rados.0.trial153.stdout:689: done (2 left) 2026-02-20T12:02:09.603 INFO:tasks.rados.rados.0.trial153.stdout:690: done (1 left) 2026-02-20T12:02:09.603 INFO:tasks.rados.rados.0.trial153.stdout:691: done (0 left) 2026-02-20T12:02:09.603 INFO:tasks.rados.rados.0.trial153.stdout:692: read oid 48 snap -1 2026-02-20T12:02:09.603 INFO:tasks.rados.rados.0.trial153.stdout:692: expect deleted 2026-02-20T12:02:09.604 INFO:tasks.rados.rados.0.trial153.stdout:693: write oid 30 current snap is 81 2026-02-20T12:02:09.604 INFO:tasks.rados.rados.0.trial153.stdout:693: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2026-02-20T12:02:09.608 INFO:tasks.rados.rados.0.trial153.stdout:693: writing trial15342161-30 from 531187 to 1310628 tid 1 2026-02-20T12:02:09.614 INFO:tasks.rados.rados.0.trial153.stdout:693: writing trial15342161-30 from 2088242 to 2826688 tid 2 2026-02-20T12:02:09.618 INFO:tasks.rados.rados.0.trial153.stdout:693: writing trial15342161-30 from 3431711 to 3731187 tid 3 2026-02-20T12:02:09.618 INFO:tasks.rados.rados.0.trial153.stdout:694: write oid 46 current snap is 81 2026-02-20T12:02:09.618 INFO:tasks.rados.rados.0.trial153.stdout:694: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2026-02-20T12:02:09.622 INFO:tasks.rados.rados.0.trial153.stdout:694: writing trial15342161-46 from 547994 to 1002684 tid 1 2026-02-20T12:02:09.626 INFO:tasks.rados.rados.0.trial153.stdout:694: writing trial15342161-46 from 1746724 to 2228828 tid 2 2026-02-20T12:02:09.630 INFO:tasks.rados.rados.0.trial153.stdout:694: writing trial15342161-46 from 2659134 to 3124280 tid 3 2026-02-20T12:02:09.633 INFO:tasks.rados.rados.0.trial153.stdout:694: writing trial15342161-46 from 3605976 to 3747994 tid 4 2026-02-20T12:02:09.633 INFO:tasks.rados.rados.0.trial153.stdout:695: write (excl) oid 37 current snap is 81 2026-02-20T12:02:09.633 INFO:tasks.rados.rados.0.trial153.stdout:695: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2026-02-20T12:02:09.637 INFO:tasks.rados.rados.0.trial153.stdout:695: writing trial15342161-37 from 564801 to 1211093 tid 1 2026-02-20T12:02:09.642 INFO:tasks.rados.rados.0.trial153.stdout:695: writing trial15342161-37 from 1921559 to 2663674 tid 2 2026-02-20T12:02:09.647 INFO:tasks.rados.rados.0.trial153.stdout:695: writing trial15342161-37 from 3202910 to 3764801 tid 3 2026-02-20T12:02:09.647 INFO:tasks.rados.rados.0.trial153.stdout:696: write (excl) oid 25 current snap is 81 2026-02-20T12:02:09.647 INFO:tasks.rados.rados.0.trial153.stdout:696: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2026-02-20T12:02:09.652 INFO:tasks.rados.rados.0.trial153.stdout:696: writing trial15342161-25 from 581608 to 1303149 tid 1 2026-02-20T12:02:09.656 INFO:tasks.rados.rados.0.trial153.stdout:696: writing trial15342161-25 from 1863688 to 2349461 tid 2 2026-02-20T12:02:09.662 INFO:tasks.rados.rados.0.trial153.stdout:696: writing trial15342161-25 from 3113980 to 3718023 tid 3 2026-02-20T12:02:09.662 INFO:tasks.rados.rados.0.trial153.stdout:697: delete oid 29 current snap is 81 2026-02-20T12:02:09.662 INFO:tasks.rados.rados.0.trial153.stdout:693: finishing write tid 1 to trial15342161-30 2026-02-20T12:02:09.662 INFO:tasks.rados.rados.0.trial153.stdout:694: finishing write tid 1 to trial15342161-46 2026-02-20T12:02:09.662 INFO:tasks.rados.rados.0.trial153.stdout:695: finishing write tid 1 to trial15342161-37 2026-02-20T12:02:09.662 INFO:tasks.rados.rados.0.trial153.stdout:693: finishing write tid 2 to trial15342161-30 2026-02-20T12:02:09.662 INFO:tasks.rados.rados.0.trial153.stdout:694: finishing write tid 2 to trial15342161-46 2026-02-20T12:02:09.667 INFO:tasks.rados.rados.0.trial153.stdout:693: finishing write tid 3 to trial15342161-30 2026-02-20T12:02:09.674 INFO:tasks.rados.rados.0.trial153.stdout:695: finishing write tid 2 to trial15342161-37 2026-02-20T12:02:09.677 INFO:tasks.rados.rados.0.trial153.stdout:694: finishing write tid 3 to trial15342161-46 2026-02-20T12:02:09.678 INFO:tasks.rados.rados.0.trial153.stdout:694: finishing write tid 4 to trial15342161-46 2026-02-20T12:02:09.679 INFO:tasks.rados.rados.0.trial153.stdout:692: done (5 left) 2026-02-20T12:02:09.679 INFO:tasks.rados.rados.0.trial153.stdout:697: done (4 left) 2026-02-20T12:02:09.680 INFO:tasks.rados.rados.0.trial153.stdout:698: delete oid 12 current snap is 81 2026-02-20T12:02:09.680 INFO:tasks.rados.rados.0.trial153.stdout:694: finishing write tid 5 to trial15342161-46 2026-02-20T12:02:09.681 INFO:tasks.rados.rados.0.trial153.stdout:693: finishing write tid 4 to trial15342161-30 2026-02-20T12:02:09.684 INFO:tasks.rados.rados.0.trial153.stdout:693: finishing write tid 5 to trial15342161-30 2026-02-20T12:02:09.684 INFO:tasks.rados.rados.0.trial153.stdout:693: oid 30 updating version 0 to 122 2026-02-20T12:02:09.684 INFO:tasks.rados.rados.0.trial153.stdout:693: oid 30 version 122 is already newer than 122 2026-02-20T12:02:09.684 INFO:tasks.rados.rados.0.trial153.stdout:693: oid 30 version 122 is already newer than 122 2026-02-20T12:02:09.684 INFO:tasks.rados.rados.0.trial153.stdout:693: oid 30 version 122 is already newer than 122 2026-02-20T12:02:09.684 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 30 v 122 (ObjNum 341 snap 81 seq_num 341) dirty exists 2026-02-20T12:02:09.684 INFO:tasks.rados.rados.0.trial153.stdout:693: left oid 30 (ObjNum 341 snap 81 seq_num 341) 2026-02-20T12:02:09.685 INFO:tasks.rados.rados.0.trial153.stdout:695: finishing write tid 3 to trial15342161-37 2026-02-20T12:02:09.686 INFO:tasks.rados.rados.0.trial153.stdout:694: finishing write tid 6 to trial15342161-46 2026-02-20T12:02:09.686 INFO:tasks.rados.rados.0.trial153.stdout:694: oid 46 updating version 0 to 44 2026-02-20T12:02:09.686 INFO:tasks.rados.rados.0.trial153.stdout:694: oid 46 version 44 is already newer than 44 2026-02-20T12:02:09.686 INFO:tasks.rados.rados.0.trial153.stdout:694: oid 46 version 44 is already newer than 44 2026-02-20T12:02:09.687 INFO:tasks.rados.rados.0.trial153.stdout:694: oid 46 version 44 is already newer than 44 2026-02-20T12:02:09.687 INFO:tasks.rados.rados.0.trial153.stdout:694: oid 46 version 44 is already newer than 44 2026-02-20T12:02:09.687 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 46 v 44 (ObjNum 342 snap 81 seq_num 342) dirty exists 2026-02-20T12:02:09.687 INFO:tasks.rados.rados.0.trial153.stdout:694: left oid 46 (ObjNum 342 snap 81 seq_num 342) 2026-02-20T12:02:09.691 INFO:tasks.rados.rados.0.trial153.stdout:695: finishing write tid 4 to trial15342161-37 2026-02-20T12:02:09.693 INFO:tasks.rados.rados.0.trial153.stdout:695: finishing write tid 5 to trial15342161-37 2026-02-20T12:02:09.693 INFO:tasks.rados.rados.0.trial153.stdout:695: oid 37 updating version 0 to 67 2026-02-20T12:02:09.694 INFO:tasks.rados.rados.0.trial153.stdout:695: oid 37 version 67 is already newer than 67 2026-02-20T12:02:09.694 INFO:tasks.rados.rados.0.trial153.stdout:695: oid 37 version 67 is already newer than 64 2026-02-20T12:02:09.694 INFO:tasks.rados.rados.0.trial153.stdout:695: oid 37 version 67 is already newer than 67 2026-02-20T12:02:09.694 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 67 (ObjNum 343 snap 81 seq_num 343) dirty exists 2026-02-20T12:02:09.694 INFO:tasks.rados.rados.0.trial153.stdout:695: left oid 37 (ObjNum 343 snap 81 seq_num 343) 2026-02-20T12:02:09.698 INFO:tasks.rados.rados.0.trial153.stdout:696: finishing write tid 1 to trial15342161-25 2026-02-20T12:02:09.707 INFO:tasks.rados.rados.0.trial153.stdout:693: done (4 left) 2026-02-20T12:02:09.707 INFO:tasks.rados.rados.0.trial153.stdout:694: done (3 left) 2026-02-20T12:02:09.707 INFO:tasks.rados.rados.0.trial153.stdout:695: done (2 left) 2026-02-20T12:02:09.707 INFO:tasks.rados.rados.0.trial153.stdout:698: done (1 left) 2026-02-20T12:02:09.707 INFO:tasks.rados.rados.0.trial153.stdout:699: snap_create 2026-02-20T12:02:09.715 INFO:tasks.rados.rados.0.trial153.stdout:696: finishing write tid 2 to trial15342161-25 2026-02-20T12:02:09.723 INFO:tasks.rados.rados.0.trial153.stdout:696: finishing write tid 3 to trial15342161-25 2026-02-20T12:02:09.727 INFO:tasks.rados.rados.0.trial153.stdout:696: finishing write tid 4 to trial15342161-25 2026-02-20T12:02:09.729 INFO:tasks.rados.rados.0.trial153.stdout:696: finishing write tid 5 to trial15342161-25 2026-02-20T12:02:09.729 INFO:tasks.rados.rados.0.trial153.stdout:696: oid 25 updating version 0 to 70 2026-02-20T12:02:09.729 INFO:tasks.rados.rados.0.trial153.stdout:696: oid 25 version 70 is already newer than 67 2026-02-20T12:02:09.729 INFO:tasks.rados.rados.0.trial153.stdout:696: oid 25 version 70 is already newer than 70 2026-02-20T12:02:09.729 INFO:tasks.rados.rados.0.trial153.stdout:696: oid 25 version 70 is already newer than 70 2026-02-20T12:02:09.729 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 25 v 70 (ObjNum 344 snap 81 seq_num 344) dirty exists 2026-02-20T12:02:09.729 INFO:tasks.rados.rados.0.trial153.stdout:696: left oid 25 (ObjNum 344 snap 81 seq_num 344) 2026-02-20T12:02:09.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~176s 2026-02-20T12:02:09.971 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:10.043 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:10.046 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:10.608 INFO:tasks.rados.rados.0.trial153.stdout:696: done (1 left) 2026-02-20T12:02:10.608 INFO:tasks.rados.rados.0.trial153.stdout:699: done (0 left) 2026-02-20T12:02:10.608 INFO:tasks.rados.rados.0.trial153.stdout:700: read oid 42 snap -1 2026-02-20T12:02:10.608 INFO:tasks.rados.rados.0.trial153.stdout:700: expect (ObjNum 280 snap 70 seq_num 280) 2026-02-20T12:02:10.608 INFO:tasks.rados.rados.0.trial153.stdout:701: delete oid 31 current snap is 82 2026-02-20T12:02:10.635 INFO:tasks.rados.rados.0.trial153.stdout:700: expect (ObjNum 280 snap 70 seq_num 280) 2026-02-20T12:02:10.692 INFO:tasks.rados.rados.0.trial153.stdout:700: done (1 left) 2026-02-20T12:02:10.692 INFO:tasks.rados.rados.0.trial153.stdout:701: done (0 left) 2026-02-20T12:02:10.692 INFO:tasks.rados.rados.0.trial153.stdout:702: snap_remove snap 80 2026-02-20T12:02:11.126 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:11.205 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:11.199+0000 7f8afb05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:11.207 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:11.203+0000 7f8afb05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:11.234 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:11.249 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:11.250 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:11.617 INFO:tasks.rados.rados.0.trial153.stdout:702: done (0 left) 2026-02-20T12:02:11.617 INFO:tasks.rados.rados.0.trial153.stdout:703: write (excl) oid 40 current snap is 82 2026-02-20T12:02:11.618 INFO:tasks.rados.rados.0.trial153.stdout:703: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2026-02-20T12:02:11.622 INFO:tasks.rados.rados.0.trial153.stdout:703: writing trial15342161-40 from 648836 to 1271373 tid 1 2026-02-20T12:02:11.627 INFO:tasks.rados.rados.0.trial153.stdout:703: writing trial15342161-40 from 1981263 to 2474374 tid 2 2026-02-20T12:02:11.633 INFO:tasks.rados.rados.0.trial153.stdout:703: writing trial15342161-40 from 3107319 to 3848836 tid 3 2026-02-20T12:02:11.633 INFO:tasks.rados.rados.0.trial153.stdout:704: snap_create 2026-02-20T12:02:11.653 INFO:tasks.rados.rados.0.trial153.stdout:703: finishing write tid 1 to trial15342161-40 2026-02-20T12:02:11.669 INFO:tasks.rados.rados.0.trial153.stdout:703: finishing write tid 2 to trial15342161-40 2026-02-20T12:02:11.675 INFO:tasks.rados.rados.0.trial153.stdout:703: finishing write tid 3 to trial15342161-40 2026-02-20T12:02:11.679 INFO:tasks.rados.rados.0.trial153.stdout:703: finishing write tid 4 to trial15342161-40 2026-02-20T12:02:11.681 INFO:tasks.rados.rados.0.trial153.stdout:703: finishing write tid 5 to trial15342161-40 2026-02-20T12:02:11.681 INFO:tasks.rados.rados.0.trial153.stdout:703: oid 40 updating version 0 to 129 2026-02-20T12:02:11.681 INFO:tasks.rados.rados.0.trial153.stdout:703: oid 40 version 129 is already newer than 129 2026-02-20T12:02:11.681 INFO:tasks.rados.rados.0.trial153.stdout:703: oid 40 version 129 is already newer than 129 2026-02-20T12:02:11.681 INFO:tasks.rados.rados.0.trial153.stdout:703: oid 40 version 129 is already newer than 126 2026-02-20T12:02:11.681 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 40 v 129 (ObjNum 348 snap 82 seq_num 348) dirty exists 2026-02-20T12:02:11.681 INFO:tasks.rados.rados.0.trial153.stdout:703: left oid 40 (ObjNum 348 snap 82 seq_num 348) 2026-02-20T12:02:12.619 INFO:tasks.rados.rados.0.trial153.stdout:703: done (1 left) 2026-02-20T12:02:12.619 INFO:tasks.rados.rados.0.trial153.stdout:704: done (0 left) 2026-02-20T12:02:12.619 INFO:tasks.rados.rados.0.trial153.stdout:705: snap_remove snap 82 2026-02-20T12:02:13.251 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:13.329 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:13.323+0000 7f017a85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:13.332 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:13.328+0000 7f017a85a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:13.358 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:13.375 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:13.375 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:13.619 INFO:tasks.rados.rados.0.trial153.stdout:705: done (0 left) 2026-02-20T12:02:13.619 INFO:tasks.rados.rados.0.trial153.stdout:706: read oid 12 snap -1 2026-02-20T12:02:13.619 INFO:tasks.rados.rados.0.trial153.stdout:706: expect deleted 2026-02-20T12:02:13.619 INFO:tasks.rados.rados.0.trial153.stdout:707: snap_create 2026-02-20T12:02:14.623 INFO:tasks.rados.rados.0.trial153.stdout:706: done (1 left) 2026-02-20T12:02:14.623 INFO:tasks.rados.rados.0.trial153.stdout:707: done (0 left) 2026-02-20T12:02:14.623 INFO:tasks.rados.rados.0.trial153.stdout:708: write (excl) oid 7 current snap is 84 2026-02-20T12:02:14.623 INFO:tasks.rados.rados.0.trial153.stdout:708: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2026-02-20T12:02:14.628 INFO:tasks.rados.rados.0.trial153.stdout:708: writing trial15342161-7 from 665643 to 1363429 tid 1 2026-02-20T12:02:14.633 INFO:tasks.rados.rados.0.trial153.stdout:708: writing trial15342161-7 from 2039745 to 2792867 tid 2 2026-02-20T12:02:14.638 INFO:tasks.rados.rados.0.trial153.stdout:708: writing trial15342161-7 from 3534742 to 3865643 tid 3 2026-02-20T12:02:14.638 INFO:tasks.rados.rados.0.trial153.stdout:709: read oid 24 snap -1 2026-02-20T12:02:14.638 INFO:tasks.rados.rados.0.trial153.stdout:709: expect (ObjNum 248 snap 63 seq_num 248) 2026-02-20T12:02:14.638 INFO:tasks.rados.rados.0.trial153.stdout:710: write (excl) oid 45 current snap is 84 2026-02-20T12:02:14.638 INFO:tasks.rados.rados.0.trial153.stdout:710: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2026-02-20T12:02:14.642 INFO:tasks.rados.rados.0.trial153.stdout:710: writing trial15342161-45 from 682450 to 1171838 tid 1 2026-02-20T12:02:14.647 INFO:tasks.rados.rados.0.trial153.stdout:710: writing trial15342161-45 from 1814580 to 2311360 tid 2 2026-02-20T12:02:14.652 INFO:tasks.rados.rados.0.trial153.stdout:710: writing trial15342161-45 from 2878518 to 3666546 tid 3 2026-02-20T12:02:14.652 INFO:tasks.rados.rados.0.trial153.stdout:711: write oid 18 current snap is 84 2026-02-20T12:02:14.652 INFO:tasks.rados.rados.0.trial153.stdout:711: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2026-02-20T12:02:14.656 INFO:tasks.rados.rados.0.trial153.stdout:711: writing trial15342161-18 from 699257 to 1263894 tid 1 2026-02-20T12:02:14.661 INFO:tasks.rados.rados.0.trial153.stdout:711: writing trial15342161-18 from 1873062 to 2513500 tid 2 2026-02-20T12:02:14.667 INFO:tasks.rados.rados.0.trial153.stdout:711: writing trial15342161-18 from 3305941 to 3899257 tid 3 2026-02-20T12:02:14.667 INFO:tasks.rados.rados.0.trial153.stdout:712: snap_remove snap 81 2026-02-20T12:02:15.047 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:15.121 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:15.124 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:15.376 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:15.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~182s 2026-02-20T12:02:15.456 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:15.450+0000 7f90823a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:15.459 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:15.454+0000 7f90823a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:15.485 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:15.503 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:15.504 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:15.629 INFO:tasks.rados.rados.0.trial153.stdout:712: done (4 left) 2026-02-20T12:02:15.629 INFO:tasks.rados.rados.0.trial153.stdout:713: snap_create 2026-02-20T12:02:15.629 INFO:tasks.rados.rados.0.trial153.stdout:708: finishing write tid 1 to trial15342161-7 2026-02-20T12:02:15.629 INFO:tasks.rados.rados.0.trial153.stdout:708: finishing write tid 2 to trial15342161-7 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:708: finishing write tid 3 to trial15342161-7 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:710: finishing write tid 1 to trial15342161-45 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:708: finishing write tid 4 to trial15342161-7 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:708: finishing write tid 5 to trial15342161-7 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:708: oid 7 updating version 0 to 73 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:708: oid 7 version 73 is already newer than 73 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:708: oid 7 version 73 is already newer than 73 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:708: oid 7 version 73 is already newer than 70 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 7 v 73 (ObjNum 349 snap 84 seq_num 349) dirty exists 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:708: left oid 7 (ObjNum 349 snap 84 seq_num 349) 2026-02-20T12:02:15.630 INFO:tasks.rados.rados.0.trial153.stdout:709: expect (ObjNum 248 snap 63 seq_num 248) 2026-02-20T12:02:15.633 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 871335 2026-02-20T12:02:15.660 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:710: finishing write tid 2 to trial15342161-45 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:710: finishing write tid 3 to trial15342161-45 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:710: finishing write tid 4 to trial15342161-45 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:710: finishing write tid 5 to trial15342161-45 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:710: oid 45 updating version 0 to 67 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:710: oid 45 version 67 is already newer than 67 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:710: oid 45 version 67 is already newer than 64 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:710: oid 45 version 67 is already newer than 67 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 67 (ObjNum 350 snap 84 seq_num 350) dirty exists 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:710: left oid 45 (ObjNum 350 snap 84 seq_num 350) 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:711: finishing write tid 1 to trial15342161-18 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:711: finishing write tid 2 to trial15342161-18 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:711: finishing write tid 3 to trial15342161-18 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:711: finishing write tid 4 to trial15342161-18 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:711: finishing write tid 5 to trial15342161-18 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:711: oid 18 updating version 0 to 49 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:711: oid 18 version 49 is already newer than 49 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:711: oid 18 version 49 is already newer than 49 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:711: oid 18 version 49 is already newer than 49 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 18 v 49 (ObjNum 351 snap 84 seq_num 351) dirty exists 2026-02-20T12:02:15.678 INFO:tasks.rados.rados.0.trial153.stdout:711: left oid 18 (ObjNum 351 snap 84 seq_num 351) 2026-02-20T12:02:16.516 INFO:tasks.rados.rados.0.trial153.stdout:708: done (4 left) 2026-02-20T12:02:16.516 INFO:tasks.rados.rados.0.trial153.stdout:709: done (3 left) 2026-02-20T12:02:16.516 INFO:tasks.rados.rados.0.trial153.stdout:710: done (2 left) 2026-02-20T12:02:16.516 INFO:tasks.rados.rados.0.trial153.stdout:711: done (1 left) 2026-02-20T12:02:16.516 INFO:tasks.rados.rados.0.trial153.stdout:713: done (0 left) 2026-02-20T12:02:16.517 INFO:tasks.rados.rados.0.trial153.stdout:714: snap_remove snap 83 2026-02-20T12:02:17.505 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:17.516 INFO:tasks.rados.rados.0.trial153.stdout:714: done (0 left) 2026-02-20T12:02:17.516 INFO:tasks.rados.rados.0.trial153.stdout:715: delete oid 19 current snap is 85 2026-02-20T12:02:17.559 INFO:tasks.rados.rados.0.trial153.stdout:715: done (0 left) 2026-02-20T12:02:17.559 INFO:tasks.rados.rados.0.trial153.stdout:716: write oid 48 current snap is 85 2026-02-20T12:02:17.559 INFO:tasks.rados.rados.0.trial153.stdout:716: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2026-02-20T12:02:17.564 INFO:tasks.rados.rados.0.trial153.stdout:716: writing trial15342161-48 from 732871 to 1448006 tid 1 2026-02-20T12:02:17.568 INFO:tasks.rados.rados.0.trial153.stdout:716: writing trial15342161-48 from 1873673 to 2517780 tid 2 2026-02-20T12:02:17.574 INFO:tasks.rados.rados.0.trial153.stdout:716: writing trial15342161-48 from 3244434 to 3932871 tid 3 2026-02-20T12:02:17.574 INFO:tasks.rados.rados.0.trial153.stdout:717: delete oid 50 current snap is 85 2026-02-20T12:02:17.590 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:17.584+0000 7f422f3dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:17.593 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:17.588+0000 7f422f3dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:17.593 INFO:tasks.rados.rados.0.trial153.stdout:716: finishing write tid 1 to trial15342161-48 2026-02-20T12:02:17.598 INFO:tasks.rados.rados.0.trial153.stdout:716: finishing write tid 2 to trial15342161-48 2026-02-20T12:02:17.605 INFO:tasks.rados.rados.0.trial153.stdout:716: finishing write tid 3 to trial15342161-48 2026-02-20T12:02:17.607 INFO:tasks.rados.rados.0.trial153.stdout:717: done (1 left) 2026-02-20T12:02:17.608 INFO:tasks.rados.rados.0.trial153.stdout:718: snap_remove snap 84 2026-02-20T12:02:17.631 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:17.646 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:18.521 INFO:tasks.rados.rados.0.trial153.stdout:718: done (1 left) 2026-02-20T12:02:18.521 INFO:tasks.rados.rados.0.trial153.stdout:719: write oid 47 current snap is 85 2026-02-20T12:02:18.521 INFO:tasks.rados.rados.0.trial153.stdout:716: finishing write tid 4 to trial15342161-48 2026-02-20T12:02:18.522 INFO:tasks.rados.rados.0.trial153.stdout:719: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2026-02-20T12:02:18.526 INFO:tasks.rados.rados.0.trial153.stdout:719: writing trial15342161-47 from 766485 to 1232118 tid 1 2026-02-20T12:02:18.532 INFO:tasks.rados.rados.0.trial153.stdout:719: writing trial15342161-47 from 1990637 to 2638413 tid 2 2026-02-20T12:02:18.537 INFO:tasks.rados.rados.0.trial153.stdout:719: writing trial15342161-47 from 3299280 to 3966485 tid 3 2026-02-20T12:02:18.537 INFO:tasks.rados.rados.0.trial153.stdout:720: write (excl) oid 38 current snap is 85 2026-02-20T12:02:18.538 INFO:tasks.rados.rados.0.trial153.stdout:716: finishing write tid 5 to trial15342161-48 2026-02-20T12:02:18.538 INFO:tasks.rados.rados.0.trial153.stdout:716: oid 48 updating version 0 to 63 2026-02-20T12:02:18.538 INFO:tasks.rados.rados.0.trial153.stdout:716: oid 48 version 63 is already newer than 63 2026-02-20T12:02:18.538 INFO:tasks.rados.rados.0.trial153.stdout:716: oid 48 version 63 is already newer than 63 2026-02-20T12:02:18.538 INFO:tasks.rados.rados.0.trial153.stdout:716: oid 48 version 63 is already newer than 63 2026-02-20T12:02:18.538 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 48 v 63 (ObjNum 353 snap 85 seq_num 353) dirty exists 2026-02-20T12:02:18.538 INFO:tasks.rados.rados.0.trial153.stdout:716: left oid 48 (ObjNum 353 snap 85 seq_num 353) 2026-02-20T12:02:18.538 INFO:tasks.rados.rados.0.trial153.stdout:720: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2026-02-20T12:02:18.542 INFO:tasks.rados.rados.0.trial153.stdout:720: writing trial15342161-38 from 783292 to 1324174 tid 1 2026-02-20T12:02:18.546 INFO:tasks.rados.rados.0.trial153.stdout:720: writing trial15342161-38 from 1932766 to 2440553 tid 2 2026-02-20T12:02:18.552 INFO:tasks.rados.rados.0.trial153.stdout:720: writing trial15342161-38 from 3210350 to 3782363 tid 3 2026-02-20T12:02:18.552 INFO:tasks.rados.rados.0.trial153.stdout:716: done (2 left) 2026-02-20T12:02:18.552 INFO:tasks.rados.rados.0.trial153.stdout:721: read oid 26 snap -1 2026-02-20T12:02:18.552 INFO:tasks.rados.rados.0.trial153.stdout:721: expect (ObjNum 332 snap 80 seq_num 332) 2026-02-20T12:02:18.552 INFO:tasks.rados.rados.0.trial153.stdout:722: write (excl) oid 49 current snap is 85 2026-02-20T12:02:18.552 INFO:tasks.rados.rados.0.trial153.stdout:722: seq_num 357 ranges {400099=616131,1591248=408851} 2026-02-20T12:02:18.555 INFO:tasks.rados.rados.0.trial153.stdout:722: writing trial15342161-49 from 400099 to 1016230 tid 1 2026-02-20T12:02:18.559 INFO:tasks.rados.rados.0.trial153.stdout:722: writing trial15342161-49 from 1591248 to 2000099 tid 2 2026-02-20T12:02:18.559 INFO:tasks.rados.rados.0.trial153.stdout:723: write oid 12 current snap is 85 2026-02-20T12:02:18.559 INFO:tasks.rados.rados.0.trial153.stdout:723: seq_num 358 ranges {416906=407733,1366083=511456} 2026-02-20T12:02:18.562 INFO:tasks.rados.rados.0.trial153.stdout:723: writing trial15342161-12 from 416906 to 824639 tid 1 2026-02-20T12:02:18.567 INFO:tasks.rados.rados.0.trial153.stdout:723: writing trial15342161-12 from 1366083 to 1877539 tid 2 2026-02-20T12:02:18.567 INFO:tasks.rados.rados.0.trial153.stdout:724: write (excl) oid 40 current snap is 85 2026-02-20T12:02:18.567 INFO:tasks.rados.rados.0.trial153.stdout:724: seq_num 359 ranges {433713=482982,1424565=609148} 2026-02-20T12:02:18.570 INFO:tasks.rados.rados.0.trial153.stdout:724: writing trial15342161-40 from 433713 to 916695 tid 1 2026-02-20T12:02:18.575 INFO:tasks.rados.rados.0.trial153.stdout:724: writing trial15342161-40 from 1424565 to 2033713 tid 2 2026-02-20T12:02:18.575 INFO:tasks.rados.rados.0.trial153.stdout:725: delete oid 10 current snap is 85 2026-02-20T12:02:18.575 INFO:tasks.rados.rados.0.trial153.stdout:720: finishing write tid 1 to trial15342161-38 2026-02-20T12:02:18.575 INFO:tasks.rados.rados.0.trial153.stdout:719: finishing write tid 1 to trial15342161-47 2026-02-20T12:02:18.584 INFO:tasks.rados.rados.0.trial153.stdout:725: done (6 left) 2026-02-20T12:02:18.584 INFO:tasks.rados.rados.0.trial153.stdout:726: snap_create 2026-02-20T12:02:18.591 INFO:tasks.rados.rados.0.trial153.stdout:719: finishing write tid 2 to trial15342161-47 2026-02-20T12:02:18.600 INFO:tasks.rados.rados.0.trial153.stdout:721: expect (ObjNum 332 snap 80 seq_num 332) 2026-02-20T12:02:18.647 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:18.730 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 3547022 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:720: finishing write tid 2 to trial15342161-38 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:719: finishing write tid 3 to trial15342161-47 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:719: finishing write tid 4 to trial15342161-47 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:720: finishing write tid 3 to trial15342161-38 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:723: finishing write tid 1 to trial15342161-12 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:719: finishing write tid 5 to trial15342161-47 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:719: oid 47 updating version 0 to 133 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:719: oid 47 version 133 is already newer than 133 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:719: oid 47 version 133 is already newer than 133 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:719: oid 47 version 133 is already newer than 133 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 47 v 133 (ObjNum 355 snap 85 seq_num 355) dirty exists 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:719: left oid 47 (ObjNum 355 snap 85 seq_num 355) 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:723: finishing write tid 2 to trial15342161-12 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:720: finishing write tid 4 to trial15342161-38 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:724: finishing write tid 1 to trial15342161-40 2026-02-20T12:02:18.731 INFO:tasks.rados.rados.0.trial153.stdout:720: finishing write tid 5 to trial15342161-38 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:720: oid 38 updating version 0 to 109 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:720: oid 38 version 109 is already newer than 106 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:720: oid 38 version 109 is already newer than 109 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:720: oid 38 version 109 is already newer than 109 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 38 v 109 (ObjNum 356 snap 85 seq_num 356) dirty exists 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:720: left oid 38 (ObjNum 356 snap 85 seq_num 356) 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:723: finishing write tid 3 to trial15342161-12 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:722: finishing write tid 1 to trial15342161-49 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:723: finishing write tid 4 to trial15342161-12 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:723: oid 12 updating version 0 to 49 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:723: oid 12 version 49 is already newer than 49 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:723: oid 12 version 49 is already newer than 49 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 12 v 49 (ObjNum 358 snap 85 seq_num 358) dirty exists 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:723: left oid 12 (ObjNum 358 snap 85 seq_num 358) 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:722: finishing write tid 2 to trial15342161-49 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:722: finishing write tid 3 to trial15342161-49 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:724: finishing write tid 2 to trial15342161-40 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:722: finishing write tid 4 to trial15342161-49 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:722: oid 49 updating version 0 to 172 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:722: oid 49 version 172 is already newer than 172 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:722: oid 49 version 172 is already newer than 170 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 49 v 172 (ObjNum 357 snap 85 seq_num 357) dirty exists 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:722: left oid 49 (ObjNum 357 snap 85 seq_num 357) 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:724: finishing write tid 3 to trial15342161-40 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:724: finishing write tid 4 to trial15342161-40 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:724: oid 40 updating version 0 to 134 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:724: oid 40 updating version 134 to 136 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:724: oid 40 version 136 is already newer than 136 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 40 v 136 (ObjNum 359 snap 85 seq_num 359) dirty exists 2026-02-20T12:02:18.732 INFO:tasks.rados.rados.0.trial153.stdout:724: left oid 40 (ObjNum 359 snap 85 seq_num 359) 2026-02-20T12:02:19.524 INFO:tasks.rados.rados.0.trial153.stdout:719: done (6 left) 2026-02-20T12:02:19.525 INFO:tasks.rados.rados.0.trial153.stdout:720: done (5 left) 2026-02-20T12:02:19.525 INFO:tasks.rados.rados.0.trial153.stdout:721: done (4 left) 2026-02-20T12:02:19.525 INFO:tasks.rados.rados.0.trial153.stdout:722: done (3 left) 2026-02-20T12:02:19.525 INFO:tasks.rados.rados.0.trial153.stdout:723: done (2 left) 2026-02-20T12:02:19.525 INFO:tasks.rados.rados.0.trial153.stdout:724: done (1 left) 2026-02-20T12:02:19.525 INFO:tasks.rados.rados.0.trial153.stdout:726: done (0 left) 2026-02-20T12:02:19.525 INFO:tasks.rados.rados.0.trial153.stdout:727: snap_create 2026-02-20T12:02:19.648 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:19.725 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:19.720+0000 7fb998e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:19.728 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:19.723+0000 7fb998e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:19.755 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:19.770 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:20.125 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:20.197 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:20.200 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:20.527 INFO:tasks.rados.rados.0.trial153.stdout:727: done (0 left) 2026-02-20T12:02:20.527 INFO:tasks.rados.rados.0.trial153.stdout:728: write (excl) oid 44 current snap is 87 2026-02-20T12:02:20.527 INFO:tasks.rados.rados.0.trial153.stdout:728: seq_num 361 ranges {467327=633480,1825176=242151} 2026-02-20T12:02:20.531 INFO:tasks.rados.rados.0.trial153.stdout:728: writing trial15342161-44 from 467327 to 1100807 tid 1 2026-02-20T12:02:20.535 INFO:tasks.rados.rados.0.trial153.stdout:728: writing trial15342161-44 from 1825176 to 2067327 tid 2 2026-02-20T12:02:20.535 INFO:tasks.rados.rados.0.trial153.stdout:729: delete oid 38 current snap is 87 2026-02-20T12:02:20.564 INFO:tasks.rados.rados.0.trial153.stdout:728: finishing write tid 1 to trial15342161-44 2026-02-20T12:02:20.568 INFO:tasks.rados.rados.0.trial153.stdout:729: done (1 left) 2026-02-20T12:02:20.569 INFO:tasks.rados.rados.0.trial153.stdout:730: write (excl) oid 13 current snap is 87 2026-02-20T12:02:20.569 INFO:tasks.rados.rados.0.trial153.stdout:730: seq_num 363 ranges {500941=783978,1942140=158801} 2026-02-20T12:02:20.574 INFO:tasks.rados.rados.0.trial153.stdout:730: writing trial15342161-13 from 500941 to 1284919 tid 1 2026-02-20T12:02:20.577 INFO:tasks.rados.rados.0.trial153.stdout:730: writing trial15342161-13 from 1942140 to 2100941 tid 2 2026-02-20T12:02:20.577 INFO:tasks.rados.rados.0.trial153.stdout:731: delete oid 17 current snap is 87 2026-02-20T12:02:20.578 INFO:tasks.rados.rados.0.trial153.stdout:728: finishing write tid 2 to trial15342161-44 2026-02-20T12:02:20.590 INFO:tasks.rados.rados.0.trial153.stdout:728: finishing write tid 3 to trial15342161-44 2026-02-20T12:02:20.591 INFO:tasks.rados.rados.0.trial153.stdout:731: done (2 left) 2026-02-20T12:02:20.592 INFO:tasks.rados.rados.0.trial153.stdout:732: write oid 18 current snap is 87 2026-02-20T12:02:20.592 INFO:tasks.rados.rados.0.trial153.stdout:732: seq_num 365 ranges {534555=650829,1659104=475451} 2026-02-20T12:02:20.596 INFO:tasks.rados.rados.0.trial153.stdout:732: writing trial15342161-18 from 534555 to 1185384 tid 1 2026-02-20T12:02:20.600 INFO:tasks.rados.rados.0.trial153.stdout:732: writing trial15342161-18 from 1659104 to 2134555 tid 2 2026-02-20T12:02:20.600 INFO:tasks.rados.rados.0.trial153.stdout:733: read oid 42 snap -1 2026-02-20T12:02:20.600 INFO:tasks.rados.rados.0.trial153.stdout:733: expect (ObjNum 280 snap 70 seq_num 280) 2026-02-20T12:02:20.601 INFO:tasks.rados.rados.0.trial153.stdout:734: write oid 28 current snap is 87 2026-02-20T12:02:20.601 INFO:tasks.rados.rados.0.trial153.stdout:730: finishing write tid 1 to trial15342161-13 2026-02-20T12:02:20.601 INFO:tasks.rados.rados.0.trial153.stdout:734: seq_num 366 ranges {551362=726078,1717586=409779} 2026-02-20T12:02:20.605 INFO:tasks.rados.rados.0.trial153.stdout:734: writing trial15342161-28 from 551362 to 1277440 tid 1 2026-02-20T12:02:20.609 INFO:tasks.rados.rados.0.trial153.stdout:734: writing trial15342161-28 from 1717586 to 2127365 tid 2 2026-02-20T12:02:20.609 INFO:tasks.rados.rados.0.trial153.stdout:735: read oid 17 snap 85 2026-02-20T12:02:20.609 INFO:tasks.rados.rados.0.trial153.stdout:735: expect deleted 2026-02-20T12:02:20.609 INFO:tasks.rados.rados.0.trial153.stdout:736: delete oid 29 current snap is 87 2026-02-20T12:02:20.609 INFO:tasks.rados.rados.0.trial153.stdout:728: finishing write tid 4 to trial15342161-44 2026-02-20T12:02:20.609 INFO:tasks.rados.rados.0.trial153.stdout:728: oid 44 updating version 0 to 126 2026-02-20T12:02:20.610 INFO:tasks.rados.rados.0.trial153.stdout:728: oid 44 version 126 is already newer than 124 2026-02-20T12:02:20.610 INFO:tasks.rados.rados.0.trial153.stdout:728: oid 44 version 126 is already newer than 126 2026-02-20T12:02:20.610 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 44 v 126 (ObjNum 361 snap 87 seq_num 361) dirty exists 2026-02-20T12:02:20.610 INFO:tasks.rados.rados.0.trial153.stdout:728: left oid 44 (ObjNum 361 snap 87 seq_num 361) 2026-02-20T12:02:20.621 INFO:tasks.rados.rados.0.trial153.stdout:730: finishing write tid 2 to trial15342161-13 2026-02-20T12:02:20.625 INFO:tasks.rados.rados.0.trial153.stdout:733: expect (ObjNum 280 snap 70 seq_num 280) 2026-02-20T12:02:20.644 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1060662 2026-02-20T12:02:20.681 INFO:tasks.rados.rados.0.trial153.stdout:734: finishing write tid 1 to trial15342161-28 2026-02-20T12:02:20.681 INFO:tasks.rados.rados.0.trial153.stdout:730: finishing write tid 3 to trial15342161-13 2026-02-20T12:02:20.681 INFO:tasks.rados.rados.0.trial153.stdout:730: finishing write tid 4 to trial15342161-13 2026-02-20T12:02:20.681 INFO:tasks.rados.rados.0.trial153.stdout:730: oid 13 updating version 0 to 242 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:730: oid 13 version 242 is already newer than 240 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:730: oid 13 version 242 is already newer than 242 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 13 v 242 (ObjNum 363 snap 87 seq_num 363) dirty exists 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:730: left oid 13 (ObjNum 363 snap 87 seq_num 363) 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:734: finishing write tid 2 to trial15342161-28 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:732: finishing write tid 1 to trial15342161-18 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:734: finishing write tid 3 to trial15342161-28 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:734: finishing write tid 4 to trial15342161-28 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:734: oid 28 updating version 0 to 139 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:734: oid 28 version 139 is already newer than 139 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:734: oid 28 version 139 is already newer than 139 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 28 v 139 (ObjNum 366 snap 87 seq_num 366) dirty exists 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:734: left oid 28 (ObjNum 366 snap 87 seq_num 366) 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:732: finishing write tid 2 to trial15342161-18 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:732: finishing write tid 3 to trial15342161-18 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:732: finishing write tid 4 to trial15342161-18 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:732: oid 18 updating version 0 to 53 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:732: oid 18 version 53 is already newer than 53 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:732: oid 18 version 53 is already newer than 53 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 18 v 53 (ObjNum 365 snap 87 seq_num 365) dirty exists 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:732: left oid 18 (ObjNum 365 snap 87 seq_num 365) 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:728: done (6 left) 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:730: done (5 left) 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:732: done (4 left) 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:733: done (3 left) 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:734: done (2 left) 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:735: done (1 left) 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:736: done (0 left) 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:737: write oid 47 current snap is 87 2026-02-20T12:02:20.682 INFO:tasks.rados.rados.0.trial153.stdout:737: seq_num 368 ranges {584976=476576,1834550=350426} 2026-02-20T12:02:20.686 INFO:tasks.rados.rados.0.trial153.stdout:737: writing trial15342161-47 from 584976 to 1061552 tid 1 2026-02-20T12:02:20.690 INFO:tasks.rados.rados.0.trial153.stdout:737: writing trial15342161-47 from 1834550 to 2184976 tid 2 2026-02-20T12:02:20.690 INFO:tasks.rados.rados.0.trial153.stdout:738: snap_remove snap 86 2026-02-20T12:02:20.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~187s 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:738: done (1 left) 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:739: delete oid 27 current snap is 87 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:737: finishing write tid 1 to trial15342161-47 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:737: finishing write tid 2 to trial15342161-47 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:737: finishing write tid 3 to trial15342161-47 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:737: finishing write tid 4 to trial15342161-47 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:737: oid 47 updating version 0 to 143 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:737: oid 47 version 143 is already newer than 143 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:737: oid 47 version 143 is already newer than 143 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 47 v 143 (ObjNum 368 snap 87 seq_num 368) dirty exists 2026-02-20T12:02:21.531 INFO:tasks.rados.rados.0.trial153.stdout:737: left oid 47 (ObjNum 368 snap 87 seq_num 368) 2026-02-20T12:02:21.575 INFO:tasks.rados.rados.0.trial153.stdout:737: done (1 left) 2026-02-20T12:02:21.576 INFO:tasks.rados.rados.0.trial153.stdout:739: done (0 left) 2026-02-20T12:02:21.576 INFO:tasks.rados.rados.0.trial153.stdout:740: snap_remove snap 85 2026-02-20T12:02:21.771 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:21.847 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:21.842+0000 7f76deddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:21.850 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:21.845+0000 7f76deddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:21.876 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:21.892 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:22.536 INFO:tasks.rados.rados.0.trial153.stdout:740: done (0 left) 2026-02-20T12:02:22.536 INFO:tasks.rados.rados.0.trial153.stdout:741: delete oid 19 current snap is 87 2026-02-20T12:02:22.567 INFO:tasks.rados.rados.0.trial153.stdout:741: done (0 left) 2026-02-20T12:02:22.567 INFO:tasks.rados.rados.0.trial153.stdout:742: snap_create 2026-02-20T12:02:23.540 INFO:tasks.rados.rados.0.trial153.stdout:742: done (0 left) 2026-02-20T12:02:23.540 INFO:tasks.rados.rados.0.trial153.stdout:743: snap_remove snap 87 2026-02-20T12:02:23.893 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:23.970 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:23.965+0000 7fe9039d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:23.973 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:23.968+0000 7fe9039d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:23.999 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:24.017 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:24.540 INFO:tasks.rados.rados.0.trial153.stdout:743: done (0 left) 2026-02-20T12:02:24.540 INFO:tasks.rados.rados.0.trial153.stdout:744: delete oid 46 current snap is 88 2026-02-20T12:02:24.580 INFO:tasks.rados.rados.0.trial153.stdout:744: done (0 left) 2026-02-20T12:02:24.580 INFO:tasks.rados.rados.0.trial153.stdout:745: write (excl) oid 37 current snap is 88 2026-02-20T12:02:24.580 INFO:tasks.rados.rados.0.trial153.stdout:745: seq_num 372 ranges {652204=777572,1952125=300079} 2026-02-20T12:02:24.586 INFO:tasks.rados.rados.0.trial153.stdout:745: writing trial15342161-37 from 652204 to 1429776 tid 1 2026-02-20T12:02:24.590 INFO:tasks.rados.rados.0.trial153.stdout:745: writing trial15342161-37 from 1952125 to 2252204 tid 2 2026-02-20T12:02:24.590 INFO:tasks.rados.rados.0.trial153.stdout:746: read oid 29 snap -1 2026-02-20T12:02:24.590 INFO:tasks.rados.rados.0.trial153.stdout:746: expect deleted 2026-02-20T12:02:24.590 INFO:tasks.rados.rados.0.trial153.stdout:747: read oid 45 snap -1 2026-02-20T12:02:24.590 INFO:tasks.rados.rados.0.trial153.stdout:747: expect (ObjNum 350 snap 84 seq_num 350) 2026-02-20T12:02:24.590 INFO:tasks.rados.rados.0.trial153.stdout:748: read oid 35 snap -1 2026-02-20T12:02:24.590 INFO:tasks.rados.rados.0.trial153.stdout:748: expect (ObjNum 335 snap 80 seq_num 335) 2026-02-20T12:02:24.590 INFO:tasks.rados.rados.0.trial153.stdout:749: read oid 33 snap -1 2026-02-20T12:02:24.591 INFO:tasks.rados.rados.0.trial153.stdout:749: expect (ObjNum 279 snap 70 seq_num 279) 2026-02-20T12:02:24.591 INFO:tasks.rados.rados.0.trial153.stdout:750: delete oid 21 current snap is 88 2026-02-20T12:02:24.605 INFO:tasks.rados.rados.0.trial153.stdout:745: finishing write tid 1 to trial15342161-37 2026-02-20T12:02:24.611 INFO:tasks.rados.rados.0.trial153.stdout:748: expect (ObjNum 335 snap 80 seq_num 335) 2026-02-20T12:02:24.690 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:24.733 INFO:tasks.rados.rados.0.trial153.stdout:747: expect (ObjNum 350 snap 84 seq_num 350) 2026-02-20T12:02:24.775 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2267537 2026-02-20T12:02:24.821 INFO:tasks.rados.rados.0.trial153.stdout:749: expect (ObjNum 279 snap 70 seq_num 279) 2026-02-20T12:02:24.841 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2158828 2026-02-20T12:02:24.848 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:24.885 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 19104 2026-02-20T12:02:24.922 INFO:tasks.rados.rados.0.trial153.stdout:745: finishing write tid 2 to trial15342161-37 2026-02-20T12:02:24.922 INFO:tasks.rados.rados.0.trial153.stdout:745: finishing write tid 3 to trial15342161-37 2026-02-20T12:02:24.923 INFO:tasks.rados.rados.0.trial153.stdout:745: finishing write tid 4 to trial15342161-37 2026-02-20T12:02:24.923 INFO:tasks.rados.rados.0.trial153.stdout:745: oid 37 updating version 0 to 70 2026-02-20T12:02:24.923 INFO:tasks.rados.rados.0.trial153.stdout:745: oid 37 version 70 is already newer than 70 2026-02-20T12:02:24.923 INFO:tasks.rados.rados.0.trial153.stdout:745: oid 37 version 70 is already newer than 68 2026-02-20T12:02:24.923 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 70 (ObjNum 372 snap 88 seq_num 372) dirty exists 2026-02-20T12:02:24.923 INFO:tasks.rados.rados.0.trial153.stdout:745: left oid 37 (ObjNum 372 snap 88 seq_num 372) 2026-02-20T12:02:24.923 INFO:tasks.rados.rados.0.trial153.stdout:745: done (5 left) 2026-02-20T12:02:24.923 INFO:tasks.rados.rados.0.trial153.stdout:746: done (4 left) 2026-02-20T12:02:24.923 INFO:tasks.rados.rados.0.trial153.stdout:747: done (3 left) 2026-02-20T12:02:24.923 INFO:tasks.rados.rados.0.trial153.stdout:748: done (2 left) 2026-02-20T12:02:24.924 INFO:tasks.rados.rados.0.trial153.stdout:749: done (1 left) 2026-02-20T12:02:24.924 INFO:tasks.rados.rados.0.trial153.stdout:750: done (0 left) 2026-02-20T12:02:24.924 INFO:tasks.rados.rados.0.trial153.stdout:751: snap_create 2026-02-20T12:02:25.201 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:25.274 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:25.278 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:25.545 INFO:tasks.rados.rados.0.trial153.stdout:751: done (0 left) 2026-02-20T12:02:25.545 INFO:tasks.rados.rados.0.trial153.stdout:752: write (excl) oid 41 current snap is 89 2026-02-20T12:02:25.546 INFO:tasks.rados.rados.0.trial153.stdout:752: seq_num 374 ranges {685818=644423,2069089=216729} 2026-02-20T12:02:25.550 INFO:tasks.rados.rados.0.trial153.stdout:752: writing trial15342161-41 from 685818 to 1330241 tid 1 2026-02-20T12:02:25.554 INFO:tasks.rados.rados.0.trial153.stdout:752: writing trial15342161-41 from 2069089 to 2285818 tid 2 2026-02-20T12:02:25.555 INFO:tasks.rados.rados.0.trial153.stdout:753: read oid 17 snap -1 2026-02-20T12:02:25.555 INFO:tasks.rados.rados.0.trial153.stdout:753: expect deleted 2026-02-20T12:02:25.555 INFO:tasks.rados.rados.0.trial153.stdout:754: snap_remove snap 88 2026-02-20T12:02:26.018 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:26.096 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:26.090+0000 7f518305a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:26.098 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:26.094+0000 7f518305a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:26.126 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:26.142 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:26.449 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~193s 2026-02-20T12:02:26.519 INFO:tasks.rados.rados.0.trial153.stdout:754: done (2 left) 2026-02-20T12:02:26.519 INFO:tasks.rados.rados.0.trial153.stdout:755: write oid 7 current snap is 89 2026-02-20T12:02:26.519 INFO:tasks.rados.rados.0.trial153.stdout:755: seq_num 375 ranges {702625=719672,2127571=175054} 2026-02-20T12:02:26.525 INFO:tasks.rados.rados.0.trial153.stdout:755: writing trial15342161-7 from 702625 to 1422297 tid 1 2026-02-20T12:02:26.529 INFO:tasks.rados.rados.0.trial153.stdout:755: writing trial15342161-7 from 2127571 to 2302625 tid 2 2026-02-20T12:02:26.529 INFO:tasks.rados.rados.0.trial153.stdout:756: write oid 15 current snap is 89 2026-02-20T12:02:26.529 INFO:tasks.rados.rados.0.trial153.stdout:756: seq_num 376 ranges {719432=794921,2186053=133379} 2026-02-20T12:02:26.535 INFO:tasks.rados.rados.0.trial153.stdout:756: writing trial15342161-15 from 719432 to 1514353 tid 1 2026-02-20T12:02:26.538 INFO:tasks.rados.rados.0.trial153.stdout:756: writing trial15342161-15 from 2186053 to 2319432 tid 2 2026-02-20T12:02:26.538 INFO:tasks.rados.rados.0.trial153.stdout:757: write oid 11 current snap is 89 2026-02-20T12:02:26.538 INFO:tasks.rados.rados.0.trial153.stdout:752: finishing write tid 1 to trial15342161-41 2026-02-20T12:02:26.538 INFO:tasks.rados.rados.0.trial153.stdout:752: finishing write tid 2 to trial15342161-41 2026-02-20T12:02:26.538 INFO:tasks.rados.rados.0.trial153.stdout:757: seq_num 377 ranges {736239=470170,1728182=571782} 2026-02-20T12:02:26.542 INFO:tasks.rados.rados.0.trial153.stdout:757: writing trial15342161-11 from 736239 to 1206409 tid 1 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:757: writing trial15342161-11 from 1728182 to 2299964 tid 2 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:753: done (4 left) 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:758: read oid 50 snap -1 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:758: expect deleted 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:759: snap_create 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:752: finishing write tid 3 to trial15342161-41 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:752: finishing write tid 4 to trial15342161-41 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:752: oid 41 updating version 0 to 58 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:752: oid 41 updating version 58 to 60 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:752: oid 41 version 60 is already newer than 60 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 41 v 60 (ObjNum 374 snap 89 seq_num 374) dirty exists 2026-02-20T12:02:26.547 INFO:tasks.rados.rados.0.trial153.stdout:752: left oid 41 (ObjNum 374 snap 89 seq_num 374) 2026-02-20T12:02:26.554 INFO:tasks.rados.rados.0.trial153.stdout:755: finishing write tid 1 to trial15342161-7 2026-02-20T12:02:26.561 INFO:tasks.rados.rados.0.trial153.stdout:756: finishing write tid 1 to trial15342161-15 2026-02-20T12:02:26.563 INFO:tasks.rados.rados.0.trial153.stdout:756: finishing write tid 2 to trial15342161-15 2026-02-20T12:02:26.566 INFO:tasks.rados.rados.0.trial153.stdout:755: finishing write tid 2 to trial15342161-7 2026-02-20T12:02:26.567 INFO:tasks.rados.rados.0.trial153.stdout:757: finishing write tid 1 to trial15342161-11 2026-02-20T12:02:26.568 INFO:tasks.rados.rados.0.trial153.stdout:756: finishing write tid 3 to trial15342161-15 2026-02-20T12:02:26.569 INFO:tasks.rados.rados.0.trial153.stdout:756: finishing write tid 4 to trial15342161-15 2026-02-20T12:02:26.569 INFO:tasks.rados.rados.0.trial153.stdout:756: oid 15 updating version 0 to 194 2026-02-20T12:02:26.569 INFO:tasks.rados.rados.0.trial153.stdout:756: oid 15 version 194 is already newer than 194 2026-02-20T12:02:26.569 INFO:tasks.rados.rados.0.trial153.stdout:756: oid 15 version 194 is already newer than 194 2026-02-20T12:02:26.569 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 15 v 194 (ObjNum 376 snap 89 seq_num 376) dirty exists 2026-02-20T12:02:26.569 INFO:tasks.rados.rados.0.trial153.stdout:756: left oid 15 (ObjNum 376 snap 89 seq_num 376) 2026-02-20T12:02:26.572 INFO:tasks.rados.rados.0.trial153.stdout:755: finishing write tid 3 to trial15342161-7 2026-02-20T12:02:26.572 INFO:tasks.rados.rados.0.trial153.stdout:757: finishing write tid 2 to trial15342161-11 2026-02-20T12:02:26.574 INFO:tasks.rados.rados.0.trial153.stdout:755: finishing write tid 4 to trial15342161-7 2026-02-20T12:02:26.574 INFO:tasks.rados.rados.0.trial153.stdout:755: oid 7 updating version 0 to 76 2026-02-20T12:02:26.574 INFO:tasks.rados.rados.0.trial153.stdout:755: oid 7 version 76 is already newer than 76 2026-02-20T12:02:26.574 INFO:tasks.rados.rados.0.trial153.stdout:755: oid 7 version 76 is already newer than 76 2026-02-20T12:02:26.574 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 7 v 76 (ObjNum 375 snap 89 seq_num 375) dirty exists 2026-02-20T12:02:26.574 INFO:tasks.rados.rados.0.trial153.stdout:755: left oid 7 (ObjNum 375 snap 89 seq_num 375) 2026-02-20T12:02:26.578 INFO:tasks.rados.rados.0.trial153.stdout:757: finishing write tid 3 to trial15342161-11 2026-02-20T12:02:26.580 INFO:tasks.rados.rados.0.trial153.stdout:757: finishing write tid 4 to trial15342161-11 2026-02-20T12:02:26.580 INFO:tasks.rados.rados.0.trial153.stdout:757: oid 11 updating version 0 to 66 2026-02-20T12:02:26.580 INFO:tasks.rados.rados.0.trial153.stdout:757: oid 11 version 66 is already newer than 66 2026-02-20T12:02:26.580 INFO:tasks.rados.rados.0.trial153.stdout:757: oid 11 version 66 is already newer than 66 2026-02-20T12:02:26.580 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 11 v 66 (ObjNum 377 snap 89 seq_num 377) dirty exists 2026-02-20T12:02:26.580 INFO:tasks.rados.rados.0.trial153.stdout:757: left oid 11 (ObjNum 377 snap 89 seq_num 377) 2026-02-20T12:02:27.553 INFO:tasks.rados.rados.0.trial153.stdout:752: done (5 left) 2026-02-20T12:02:27.553 INFO:tasks.rados.rados.0.trial153.stdout:755: done (4 left) 2026-02-20T12:02:27.553 INFO:tasks.rados.rados.0.trial153.stdout:756: done (3 left) 2026-02-20T12:02:27.553 INFO:tasks.rados.rados.0.trial153.stdout:757: done (2 left) 2026-02-20T12:02:27.553 INFO:tasks.rados.rados.0.trial153.stdout:758: done (1 left) 2026-02-20T12:02:27.553 INFO:tasks.rados.rados.0.trial153.stdout:759: done (0 left) 2026-02-20T12:02:27.553 INFO:tasks.rados.rados.0.trial153.stdout:760: write oid 35 current snap is 90 2026-02-20T12:02:27.553 INFO:tasks.rados.rados.0.trial153.stdout:760: seq_num 378 ranges {753046=545419,1786664=431793} 2026-02-20T12:02:27.557 INFO:tasks.rados.rados.0.trial153.stdout:760: writing trial15342161-35 from 753046 to 1298465 tid 1 2026-02-20T12:02:27.561 INFO:tasks.rados.rados.0.trial153.stdout:760: writing trial15342161-35 from 1786664 to 2218457 tid 2 2026-02-20T12:02:27.561 INFO:tasks.rados.rados.0.trial153.stdout:761: snap_create 2026-02-20T12:02:27.592 INFO:tasks.rados.rados.0.trial153.stdout:760: finishing write tid 1 to trial15342161-35 2026-02-20T12:02:27.600 INFO:tasks.rados.rados.0.trial153.stdout:760: finishing write tid 2 to trial15342161-35 2026-02-20T12:02:27.606 INFO:tasks.rados.rados.0.trial153.stdout:760: finishing write tid 3 to trial15342161-35 2026-02-20T12:02:27.608 INFO:tasks.rados.rados.0.trial153.stdout:760: finishing write tid 4 to trial15342161-35 2026-02-20T12:02:27.608 INFO:tasks.rados.rados.0.trial153.stdout:760: oid 35 updating version 0 to 147 2026-02-20T12:02:27.608 INFO:tasks.rados.rados.0.trial153.stdout:760: oid 35 version 147 is already newer than 147 2026-02-20T12:02:27.608 INFO:tasks.rados.rados.0.trial153.stdout:760: oid 35 version 147 is already newer than 147 2026-02-20T12:02:27.608 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 35 v 147 (ObjNum 378 snap 90 seq_num 378) dirty exists 2026-02-20T12:02:27.608 INFO:tasks.rados.rados.0.trial153.stdout:760: left oid 35 (ObjNum 378 snap 90 seq_num 378) 2026-02-20T12:02:28.143 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:28.221 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:28.215+0000 7ff95f783640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:28.224 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:28.219+0000 7ff95f783640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:28.250 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:28.268 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:28.556 INFO:tasks.rados.rados.0.trial153.stdout:760: done (1 left) 2026-02-20T12:02:28.556 INFO:tasks.rados.rados.0.trial153.stdout:761: done (0 left) 2026-02-20T12:02:28.556 INFO:tasks.rados.rados.0.trial153.stdout:762: write (excl) oid 7 current snap is 91 2026-02-20T12:02:28.556 INFO:tasks.rados.rados.0.trial153.stdout:762: seq_num 379 ranges {769853=620668,1845146=524707} 2026-02-20T12:02:28.560 INFO:tasks.rados.rados.0.trial153.stdout:762: writing trial15342161-7 from 769853 to 1390521 tid 1 2026-02-20T12:02:28.564 INFO:tasks.rados.rados.0.trial153.stdout:762: writing trial15342161-7 from 1845146 to 2369853 tid 2 2026-02-20T12:02:28.565 INFO:tasks.rados.rados.0.trial153.stdout:763: snap_remove snap 89 2026-02-20T12:02:29.560 INFO:tasks.rados.rados.0.trial153.stdout:763: done (1 left) 2026-02-20T12:02:29.560 INFO:tasks.rados.rados.0.trial153.stdout:764: snap_remove snap 90 2026-02-20T12:02:30.269 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:30.279 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:30.346 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:30.340+0000 7f89b40cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:30.349 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:30.344+0000 7f89b40cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:30.350 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:30.353 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:30.376 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:30.391 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:30.391 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:30.565 INFO:tasks.rados.rados.0.trial153.stdout:764: done (1 left) 2026-02-20T12:02:30.565 INFO:tasks.rados.rados.0.trial153.stdout:765: snap_create 2026-02-20T12:02:30.566 INFO:tasks.rados.rados.0.trial153.stdout:762: finishing write tid 1 to trial15342161-7 2026-02-20T12:02:30.566 INFO:tasks.rados.rados.0.trial153.stdout:762: finishing write tid 2 to trial15342161-7 2026-02-20T12:02:30.566 INFO:tasks.rados.rados.0.trial153.stdout:762: finishing write tid 3 to trial15342161-7 2026-02-20T12:02:30.566 INFO:tasks.rados.rados.0.trial153.stdout:762: finishing write tid 4 to trial15342161-7 2026-02-20T12:02:30.566 INFO:tasks.rados.rados.0.trial153.stdout:762: oid 7 updating version 0 to 80 2026-02-20T12:02:30.566 INFO:tasks.rados.rados.0.trial153.stdout:762: oid 7 version 80 is already newer than 78 2026-02-20T12:02:30.567 INFO:tasks.rados.rados.0.trial153.stdout:762: oid 7 version 80 is already newer than 80 2026-02-20T12:02:30.567 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 7 v 80 (ObjNum 379 snap 91 seq_num 379) dirty exists 2026-02-20T12:02:30.567 INFO:tasks.rados.rados.0.trial153.stdout:762: left oid 7 (ObjNum 379 snap 91 seq_num 379) 2026-02-20T12:02:31.566 INFO:tasks.rados.rados.0.trial153.stdout:762: done (1 left) 2026-02-20T12:02:31.566 INFO:tasks.rados.rados.0.trial153.stdout:765: done (0 left) 2026-02-20T12:02:31.566 INFO:tasks.rados.rados.0.trial153.stdout:766: read oid 26 snap -1 2026-02-20T12:02:31.566 INFO:tasks.rados.rados.0.trial153.stdout:766: expect (ObjNum 332 snap 80 seq_num 332) 2026-02-20T12:02:31.566 INFO:tasks.rados.rados.0.trial153.stdout:767: write oid 44 current snap is 92 2026-02-20T12:02:31.566 INFO:tasks.rados.rados.0.trial153.stdout:767: seq_num 380 ranges {786660=695917,1903628=435462} 2026-02-20T12:02:31.571 INFO:tasks.rados.rados.0.trial153.stdout:767: writing trial15342161-44 from 786660 to 1482577 tid 1 2026-02-20T12:02:31.575 INFO:tasks.rados.rados.0.trial153.stdout:767: writing trial15342161-44 from 1903628 to 2339090 tid 2 2026-02-20T12:02:31.575 INFO:tasks.rados.rados.0.trial153.stdout:768: read oid 5 snap -1 2026-02-20T12:02:31.575 INFO:tasks.rados.rados.0.trial153.stdout:768: expect (ObjNum 312 snap 77 seq_num 312) 2026-02-20T12:02:31.575 INFO:tasks.rados.rados.0.trial153.stdout:769: read oid 45 snap -1 2026-02-20T12:02:31.576 INFO:tasks.rados.rados.0.trial153.stdout:769: expect (ObjNum 350 snap 84 seq_num 350) 2026-02-20T12:02:31.576 INFO:tasks.rados.rados.0.trial153.stdout:770: write oid 38 current snap is 92 2026-02-20T12:02:31.576 INFO:tasks.rados.rados.0.trial153.stdout:770: seq_num 381 ranges {403467=487519,1562110=579120} 2026-02-20T12:02:31.578 INFO:tasks.rados.rados.0.trial153.stdout:770: writing trial15342161-38 from 403467 to 890986 tid 1 2026-02-20T12:02:31.584 INFO:tasks.rados.rados.0.trial153.stdout:770: writing trial15342161-38 from 1562110 to 2141230 tid 2 2026-02-20T12:02:31.584 INFO:tasks.rados.rados.0.trial153.stdout:771: read oid 13 snap -1 2026-02-20T12:02:31.584 INFO:tasks.rados.rados.0.trial153.stdout:771: expect (ObjNum 363 snap 87 seq_num 363) 2026-02-20T12:02:31.584 INFO:tasks.rados.rados.0.trial153.stdout:772: read oid 39 snap 91 2026-02-20T12:02:31.584 INFO:tasks.rados.rados.0.trial153.stdout:772: expect deleted 2026-02-20T12:02:31.584 INFO:tasks.rados.rados.0.trial153.stdout:773: delete oid 24 current snap is 92 2026-02-20T12:02:31.597 INFO:tasks.rados.rados.0.trial153.stdout:768: expect (ObjNum 312 snap 77 seq_num 312) 2026-02-20T12:02:31.636 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 3047935 2026-02-20T12:02:31.665 INFO:tasks.rados.rados.0.trial153.stdout:768: done (7 left) 2026-02-20T12:02:31.665 INFO:tasks.rados.rados.0.trial153.stdout:773: done (6 left) 2026-02-20T12:02:31.665 INFO:tasks.rados.rados.0.trial153.stdout:774: delete oid 6 current snap is 92 2026-02-20T12:02:31.666 INFO:tasks.rados.rados.0.trial153.stdout:770: finishing write tid 1 to trial15342161-38 2026-02-20T12:02:31.666 INFO:tasks.rados.rados.0.trial153.stdout:771: expect (ObjNum 363 snap 87 seq_num 363) 2026-02-20T12:02:31.669 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1160182 2026-02-20T12:02:31.678 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:31.708 INFO:tasks.rados.rados.0.trial153.stdout:771: done (6 left) 2026-02-20T12:02:31.708 INFO:tasks.rados.rados.0.trial153.stdout:774: done (5 left) 2026-02-20T12:02:31.708 INFO:tasks.rados.rados.0.trial153.stdout:775: delete oid 19 current snap is 92 2026-02-20T12:02:31.708 INFO:tasks.rados.rados.0.trial153.stdout:766: expect (ObjNum 332 snap 80 seq_num 332) 2026-02-20T12:02:31.709 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:31.757 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:31.845 INFO:tasks.rados.rados.0.trial153.stdout:769: expect (ObjNum 350 snap 84 seq_num 350) 2026-02-20T12:02:31.928 INFO:tasks.rados.rados.0.trial153.stdout:767: finishing write tid 1 to trial15342161-44 2026-02-20T12:02:31.928 INFO:tasks.rados.rados.0.trial153.stdout:770: finishing write tid 2 to trial15342161-38 2026-02-20T12:02:31.928 INFO:tasks.rados.rados.0.trial153.stdout:770: finishing write tid 3 to trial15342161-38 2026-02-20T12:02:31.928 INFO:tasks.rados.rados.0.trial153.stdout:770: finishing write tid 4 to trial15342161-38 2026-02-20T12:02:31.928 INFO:tasks.rados.rados.0.trial153.stdout:770: oid 38 updating version 0 to 114 2026-02-20T12:02:31.928 INFO:tasks.rados.rados.0.trial153.stdout:770: oid 38 version 114 is already newer than 114 2026-02-20T12:02:31.928 INFO:tasks.rados.rados.0.trial153.stdout:770: oid 38 version 114 is already newer than 114 2026-02-20T12:02:31.928 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 38 v 114 (ObjNum 381 snap 92 seq_num 381) dirty exists 2026-02-20T12:02:31.928 INFO:tasks.rados.rados.0.trial153.stdout:770: left oid 38 (ObjNum 381 snap 92 seq_num 381) 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:767: finishing write tid 2 to trial15342161-44 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:767: finishing write tid 3 to trial15342161-44 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:766: done (5 left) 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:769: done (4 left) 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:770: done (3 left) 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:772: done (2 left) 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:775: done (1 left) 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:776: delete oid 40 current snap is 92 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:767: finishing write tid 4 to trial15342161-44 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:767: oid 44 updating version 0 to 131 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:767: oid 44 version 131 is already newer than 131 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:767: oid 44 version 131 is already newer than 131 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 44 v 131 (ObjNum 380 snap 92 seq_num 380) dirty exists 2026-02-20T12:02:31.929 INFO:tasks.rados.rados.0.trial153.stdout:767: left oid 44 (ObjNum 380 snap 92 seq_num 380) 2026-02-20T12:02:31.952 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~198s 2026-02-20T12:02:31.953 INFO:tasks.rados.rados.0.trial153.stdout:767: done (1 left) 2026-02-20T12:02:31.953 INFO:tasks.rados.rados.0.trial153.stdout:776: done (0 left) 2026-02-20T12:02:31.953 INFO:tasks.rados.rados.0.trial153.stdout:777: delete oid 44 current snap is 92 2026-02-20T12:02:31.966 INFO:tasks.rados.rados.0.trial153.stdout:777: done (0 left) 2026-02-20T12:02:31.966 INFO:tasks.rados.rados.0.trial153.stdout:778: write (excl) oid 14 current snap is 92 2026-02-20T12:02:31.966 INFO:tasks.rados.rados.0.trial153.stdout:778: seq_num 387 ranges {504309=539013,1796649=590127} 2026-02-20T12:02:31.969 INFO:tasks.rados.rados.0.trial153.stdout:778: writing trial15342161-14 from 504309 to 1043322 tid 1 2026-02-20T12:02:31.975 INFO:tasks.rados.rados.0.trial153.stdout:778: writing trial15342161-14 from 1796649 to 2386776 tid 2 2026-02-20T12:02:31.975 INFO:tasks.rados.rados.0.trial153.stdout:779: snap_create 2026-02-20T12:02:31.994 INFO:tasks.rados.rados.0.trial153.stdout:778: finishing write tid 1 to trial15342161-14 2026-02-20T12:02:32.007 INFO:tasks.rados.rados.0.trial153.stdout:778: finishing write tid 2 to trial15342161-14 2026-02-20T12:02:32.013 INFO:tasks.rados.rados.0.trial153.stdout:778: finishing write tid 3 to trial15342161-14 2026-02-20T12:02:32.019 INFO:tasks.rados.rados.0.trial153.stdout:778: finishing write tid 4 to trial15342161-14 2026-02-20T12:02:32.020 INFO:tasks.rados.rados.0.trial153.stdout:778: oid 14 updating version 0 to 201 2026-02-20T12:02:32.020 INFO:tasks.rados.rados.0.trial153.stdout:778: oid 14 version 201 is already newer than 201 2026-02-20T12:02:32.020 INFO:tasks.rados.rados.0.trial153.stdout:778: oid 14 version 201 is already newer than 199 2026-02-20T12:02:32.020 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 201 (ObjNum 387 snap 92 seq_num 387) dirty exists 2026-02-20T12:02:32.020 INFO:tasks.rados.rados.0.trial153.stdout:778: left oid 14 (ObjNum 387 snap 92 seq_num 387) 2026-02-20T12:02:32.392 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:32.469 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:32.463+0000 7f457f0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:32.471 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:32.467+0000 7f457f0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:32.499 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:32.515 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:32.570 INFO:tasks.rados.rados.0.trial153.stdout:778: done (1 left) 2026-02-20T12:02:32.570 INFO:tasks.rados.rados.0.trial153.stdout:779: done (0 left) 2026-02-20T12:02:32.570 INFO:tasks.rados.rados.0.trial153.stdout:780: delete oid 1 current snap is 93 2026-02-20T12:02:32.608 INFO:tasks.rados.rados.0.trial153.stdout:780: done (0 left) 2026-02-20T12:02:32.608 INFO:tasks.rados.rados.0.trial153.stdout:781: write oid 25 current snap is 93 2026-02-20T12:02:32.609 INFO:tasks.rados.rados.0.trial153.stdout:781: seq_num 389 ranges {537923=405864,1513613=593796} 2026-02-20T12:02:32.611 INFO:tasks.rados.rados.0.trial153.stdout:781: writing trial15342161-25 from 537923 to 943787 tid 1 2026-02-20T12:02:32.616 INFO:tasks.rados.rados.0.trial153.stdout:781: writing trial15342161-25 from 1513613 to 2107409 tid 2 2026-02-20T12:02:32.616 INFO:tasks.rados.rados.0.trial153.stdout:782: write oid 37 current snap is 93 2026-02-20T12:02:32.616 INFO:tasks.rados.rados.0.trial153.stdout:782: seq_num 390 ranges {554730=481113,1572095=737454} 2026-02-20T12:02:32.620 INFO:tasks.rados.rados.0.trial153.stdout:782: writing trial15342161-37 from 554730 to 1035843 tid 1 2026-02-20T12:02:32.625 INFO:tasks.rados.rados.0.trial153.stdout:782: writing trial15342161-37 from 1572095 to 2309549 tid 2 2026-02-20T12:02:32.625 INFO:tasks.rados.rados.0.trial153.stdout:783: snap_create 2026-02-20T12:02:32.646 INFO:tasks.rados.rados.0.trial153.stdout:781: finishing write tid 1 to trial15342161-25 2026-02-20T12:02:32.653 INFO:tasks.rados.rados.0.trial153.stdout:781: finishing write tid 2 to trial15342161-25 2026-02-20T12:02:32.654 INFO:tasks.rados.rados.0.trial153.stdout:782: finishing write tid 1 to trial15342161-37 2026-02-20T12:02:32.659 INFO:tasks.rados.rados.0.trial153.stdout:781: finishing write tid 3 to trial15342161-25 2026-02-20T12:02:32.661 INFO:tasks.rados.rados.0.trial153.stdout:781: finishing write tid 4 to trial15342161-25 2026-02-20T12:02:32.661 INFO:tasks.rados.rados.0.trial153.stdout:781: oid 25 updating version 0 to 76 2026-02-20T12:02:32.661 INFO:tasks.rados.rados.0.trial153.stdout:781: oid 25 version 76 is already newer than 76 2026-02-20T12:02:32.661 INFO:tasks.rados.rados.0.trial153.stdout:781: oid 25 version 76 is already newer than 76 2026-02-20T12:02:32.661 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 25 v 76 (ObjNum 389 snap 93 seq_num 389) dirty exists 2026-02-20T12:02:32.661 INFO:tasks.rados.rados.0.trial153.stdout:781: left oid 25 (ObjNum 389 snap 93 seq_num 389) 2026-02-20T12:02:32.662 INFO:tasks.rados.rados.0.trial153.stdout:782: finishing write tid 2 to trial15342161-37 2026-02-20T12:02:32.666 INFO:tasks.rados.rados.0.trial153.stdout:782: finishing write tid 3 to trial15342161-37 2026-02-20T12:02:32.668 INFO:tasks.rados.rados.0.trial153.stdout:782: finishing write tid 4 to trial15342161-37 2026-02-20T12:02:32.668 INFO:tasks.rados.rados.0.trial153.stdout:782: oid 37 updating version 0 to 74 2026-02-20T12:02:32.668 INFO:tasks.rados.rados.0.trial153.stdout:782: oid 37 version 74 is already newer than 74 2026-02-20T12:02:32.668 INFO:tasks.rados.rados.0.trial153.stdout:782: oid 37 version 74 is already newer than 74 2026-02-20T12:02:32.668 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 74 (ObjNum 390 snap 93 seq_num 390) dirty exists 2026-02-20T12:02:32.669 INFO:tasks.rados.rados.0.trial153.stdout:782: left oid 37 (ObjNum 390 snap 93 seq_num 390) 2026-02-20T12:02:33.575 INFO:tasks.rados.rados.0.trial153.stdout:781: done (2 left) 2026-02-20T12:02:33.576 INFO:tasks.rados.rados.0.trial153.stdout:782: done (1 left) 2026-02-20T12:02:33.576 INFO:tasks.rados.rados.0.trial153.stdout:783: done (0 left) 2026-02-20T12:02:33.576 INFO:tasks.rados.rados.0.trial153.stdout:784: write oid 23 current snap is 94 2026-02-20T12:02:33.576 INFO:tasks.rados.rados.0.trial153.stdout:784: seq_num 391 ranges {571537=556362,1630577=597465} 2026-02-20T12:02:33.579 INFO:tasks.rados.rados.0.trial153.stdout:784: writing trial15342161-23 from 571537 to 1127899 tid 1 2026-02-20T12:02:33.584 INFO:tasks.rados.rados.0.trial153.stdout:784: writing trial15342161-23 from 1630577 to 2228042 tid 2 2026-02-20T12:02:33.584 INFO:tasks.rados.rados.0.trial153.stdout:785: snap_create 2026-02-20T12:02:33.610 INFO:tasks.rados.rados.0.trial153.stdout:784: finishing write tid 1 to trial15342161-23 2026-02-20T12:02:33.616 INFO:tasks.rados.rados.0.trial153.stdout:784: finishing write tid 2 to trial15342161-23 2026-02-20T12:02:33.620 INFO:tasks.rados.rados.0.trial153.stdout:784: finishing write tid 3 to trial15342161-23 2026-02-20T12:02:33.622 INFO:tasks.rados.rados.0.trial153.stdout:784: finishing write tid 4 to trial15342161-23 2026-02-20T12:02:33.622 INFO:tasks.rados.rados.0.trial153.stdout:784: oid 23 updating version 0 to 152 2026-02-20T12:02:33.622 INFO:tasks.rados.rados.0.trial153.stdout:784: oid 23 version 152 is already newer than 152 2026-02-20T12:02:33.622 INFO:tasks.rados.rados.0.trial153.stdout:784: oid 23 version 152 is already newer than 152 2026-02-20T12:02:33.622 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 23 v 152 (ObjNum 391 snap 94 seq_num 391) dirty exists 2026-02-20T12:02:33.622 INFO:tasks.rados.rados.0.trial153.stdout:784: left oid 23 (ObjNum 391 snap 94 seq_num 391) 2026-02-20T12:02:34.517 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:34.578 INFO:tasks.rados.rados.0.trial153.stdout:784: done (1 left) 2026-02-20T12:02:34.578 INFO:tasks.rados.rados.0.trial153.stdout:785: done (0 left) 2026-02-20T12:02:34.578 INFO:tasks.rados.rados.0.trial153.stdout:786: read oid 17 snap -1 2026-02-20T12:02:34.579 INFO:tasks.rados.rados.0.trial153.stdout:786: expect deleted 2026-02-20T12:02:34.579 INFO:tasks.rados.rados.0.trial153.stdout:787: read oid 43 snap -1 2026-02-20T12:02:34.579 INFO:tasks.rados.rados.0.trial153.stdout:787: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:02:34.579 INFO:tasks.rados.rados.0.trial153.stdout:788: read oid 27 snap -1 2026-02-20T12:02:34.579 INFO:tasks.rados.rados.0.trial153.stdout:788: expect deleted 2026-02-20T12:02:34.579 INFO:tasks.rados.rados.0.trial153.stdout:789: read oid 11 snap -1 2026-02-20T12:02:34.579 INFO:tasks.rados.rados.0.trial153.stdout:789: expect (ObjNum 377 snap 89 seq_num 377) 2026-02-20T12:02:34.579 INFO:tasks.rados.rados.0.trial153.stdout:790: read oid 9 snap -1 2026-02-20T12:02:34.579 INFO:tasks.rados.rados.0.trial153.stdout:790: expect (ObjNum 260 snap 68 seq_num 260) 2026-02-20T12:02:34.579 INFO:tasks.rados.rados.0.trial153.stdout:791: write oid 44 current snap is 95 2026-02-20T12:02:34.580 INFO:tasks.rados.rados.0.trial153.stdout:791: seq_num 392 ranges {588344=631611,1689059=741123} 2026-02-20T12:02:34.582 INFO:tasks.rados.rados.0.trial153.stdout:791: writing trial15342161-44 from 588344 to 1219955 tid 1 2026-02-20T12:02:34.587 INFO:tasks.rados.rados.0.trial153.stdout:791: writing trial15342161-44 from 1689059 to 2430182 tid 2 2026-02-20T12:02:34.587 INFO:tasks.rados.rados.0.trial153.stdout:792: delete oid 14 current snap is 95 2026-02-20T12:02:34.597 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:34.591+0000 7fd735ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:34.599 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:34.595+0000 7fd735ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:34.607 INFO:tasks.rados.rados.0.trial153.stdout:789: expect (ObjNum 377 snap 89 seq_num 377) 2026-02-20T12:02:34.619 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:34.640 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:34.659 INFO:tasks.rados.rados.0.trial153.stdout:790: expect (ObjNum 260 snap 68 seq_num 260) 2026-02-20T12:02:34.659 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:34.674 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:34.674 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:34.708 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1536569 2026-02-20T12:02:34.756 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1436773 2026-02-20T12:02:34.792 INFO:tasks.rados.rados.0.trial153.stdout:787: expect (ObjNum 192 snap 47 seq_num 192) 2026-02-20T12:02:34.792 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:34.855 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2154147 2026-02-20T12:02:34.920 INFO:tasks.rados.rados.0.trial153.stdout:791: finishing write tid 1 to trial15342161-44 2026-02-20T12:02:34.920 INFO:tasks.rados.rados.0.trial153.stdout:786: done (6 left) 2026-02-20T12:02:34.920 INFO:tasks.rados.rados.0.trial153.stdout:787: done (5 left) 2026-02-20T12:02:34.920 INFO:tasks.rados.rados.0.trial153.stdout:788: done (4 left) 2026-02-20T12:02:34.920 INFO:tasks.rados.rados.0.trial153.stdout:789: done (3 left) 2026-02-20T12:02:34.921 INFO:tasks.rados.rados.0.trial153.stdout:790: done (2 left) 2026-02-20T12:02:34.921 INFO:tasks.rados.rados.0.trial153.stdout:792: done (1 left) 2026-02-20T12:02:34.921 INFO:tasks.rados.rados.0.trial153.stdout:793: write oid 17 current snap is 95 2026-02-20T12:02:34.921 INFO:tasks.rados.rados.0.trial153.stdout:791: finishing write tid 2 to trial15342161-44 2026-02-20T12:02:34.921 INFO:tasks.rados.rados.0.trial153.stdout:791: finishing write tid 3 to trial15342161-44 2026-02-20T12:02:34.921 INFO:tasks.rados.rados.0.trial153.stdout:793: seq_num 394 ranges {621958=782109,2089670=532288} 2026-02-20T12:02:34.925 INFO:tasks.rados.rados.0.trial153.stdout:793: writing trial15342161-17 from 621958 to 1404067 tid 1 2026-02-20T12:02:34.930 INFO:tasks.rados.rados.0.trial153.stdout:793: writing trial15342161-17 from 2089670 to 2621958 tid 2 2026-02-20T12:02:34.930 INFO:tasks.rados.rados.0.trial153.stdout:794: write (excl) oid 5 current snap is 95 2026-02-20T12:02:34.930 INFO:tasks.rados.rados.0.trial153.stdout:794: seq_num 395 ranges {638765=457358,1748152=604803} 2026-02-20T12:02:34.934 INFO:tasks.rados.rados.0.trial153.stdout:794: writing trial15342161-5 from 638765 to 1096123 tid 1 2026-02-20T12:02:34.940 INFO:tasks.rados.rados.0.trial153.stdout:794: writing trial15342161-5 from 1748152 to 2352955 tid 2 2026-02-20T12:02:34.940 INFO:tasks.rados.rados.0.trial153.stdout:795: read oid 22 snap 91 2026-02-20T12:02:34.940 INFO:tasks.rados.rados.0.trial153.stdout:795: expect (ObjNum 319 snap 78 seq_num 319) 2026-02-20T12:02:34.940 INFO:tasks.rados.rados.0.trial153.stdout:796: read oid 31 snap -1 2026-02-20T12:02:34.940 INFO:tasks.rados.rados.0.trial153.stdout:796: expect deleted 2026-02-20T12:02:34.940 INFO:tasks.rados.rados.0.trial153.stdout:797: write oid 45 current snap is 95 2026-02-20T12:02:34.940 INFO:tasks.rados.rados.0.trial153.stdout:791: finishing write tid 4 to trial15342161-44 2026-02-20T12:02:34.940 INFO:tasks.rados.rados.0.trial153.stdout:791: oid 44 updating version 0 to 135 2026-02-20T12:02:34.940 INFO:tasks.rados.rados.0.trial153.stdout:791: oid 44 version 135 is already newer than 135 2026-02-20T12:02:34.941 INFO:tasks.rados.rados.0.trial153.stdout:791: oid 44 version 135 is already newer than 135 2026-02-20T12:02:34.941 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 44 v 135 (ObjNum 392 snap 95 seq_num 392) dirty exists 2026-02-20T12:02:34.941 INFO:tasks.rados.rados.0.trial153.stdout:791: left oid 44 (ObjNum 392 snap 95 seq_num 392) 2026-02-20T12:02:34.941 INFO:tasks.rados.rados.0.trial153.stdout:797: seq_num 396 ranges {655572=648960,1922987=732585} 2026-02-20T12:02:34.944 INFO:tasks.rados.rados.0.trial153.stdout:797: writing trial15342161-45 from 655572 to 1304532 tid 1 2026-02-20T12:02:34.950 INFO:tasks.rados.rados.0.trial153.stdout:797: writing trial15342161-45 from 1922987 to 2655572 tid 2 2026-02-20T12:02:34.950 INFO:tasks.rados.rados.0.trial153.stdout:791: done (5 left) 2026-02-20T12:02:34.950 INFO:tasks.rados.rados.0.trial153.stdout:798: read oid 13 snap -1 2026-02-20T12:02:34.950 INFO:tasks.rados.rados.0.trial153.stdout:798: expect (ObjNum 363 snap 87 seq_num 363) 2026-02-20T12:02:34.950 INFO:tasks.rados.rados.0.trial153.stdout:799: snap_remove snap 92 2026-02-20T12:02:35.355 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:35.427 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:35.430 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:35.583 INFO:tasks.rados.rados.0.trial153.stdout:799: done (6 left) 2026-02-20T12:02:35.583 INFO:tasks.rados.rados.0.trial153.stdout:800: snap_create 2026-02-20T12:02:35.583 INFO:tasks.rados.rados.0.trial153.stdout:793: finishing write tid 1 to trial15342161-17 2026-02-20T12:02:35.583 INFO:tasks.rados.rados.0.trial153.stdout:793: finishing write tid 2 to trial15342161-17 2026-02-20T12:02:35.583 INFO:tasks.rados.rados.0.trial153.stdout:795: expect (ObjNum 319 snap 78 seq_num 319) 2026-02-20T12:02:35.612 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:35.640 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 392100 2026-02-20T12:02:35.666 INFO:tasks.rados.rados.0.trial153.stdout:793: finishing write tid 3 to trial15342161-17 2026-02-20T12:02:35.666 INFO:tasks.rados.rados.0.trial153.stdout:793: finishing write tid 4 to trial15342161-17 2026-02-20T12:02:35.666 INFO:tasks.rados.rados.0.trial153.stdout:793: oid 17 updating version 0 to 176 2026-02-20T12:02:35.666 INFO:tasks.rados.rados.0.trial153.stdout:793: oid 17 version 176 is already newer than 176 2026-02-20T12:02:35.666 INFO:tasks.rados.rados.0.trial153.stdout:793: oid 17 version 176 is already newer than 176 2026-02-20T12:02:35.666 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 17 v 176 (ObjNum 394 snap 95 seq_num 394) dirty exists 2026-02-20T12:02:35.666 INFO:tasks.rados.rados.0.trial153.stdout:793: left oid 17 (ObjNum 394 snap 95 seq_num 394) 2026-02-20T12:02:35.666 INFO:tasks.rados.rados.0.trial153.stdout:797: finishing write tid 1 to trial15342161-45 2026-02-20T12:02:35.666 INFO:tasks.rados.rados.0.trial153.stdout:798: expect (ObjNum 363 snap 87 seq_num 363) 2026-02-20T12:02:35.677 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:35.695 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:35.713 INFO:tasks.rados.rados.0.trial153.stdout:794: finishing write tid 1 to trial15342161-5 2026-02-20T12:02:35.713 INFO:tasks.rados.rados.0.trial153.stdout:797: finishing write tid 2 to trial15342161-45 2026-02-20T12:02:35.713 INFO:tasks.rados.rados.0.trial153.stdout:797: finishing write tid 3 to trial15342161-45 2026-02-20T12:02:35.713 INFO:tasks.rados.rados.0.trial153.stdout:797: finishing write tid 4 to trial15342161-45 2026-02-20T12:02:35.713 INFO:tasks.rados.rados.0.trial153.stdout:797: oid 45 updating version 0 to 71 2026-02-20T12:02:35.713 INFO:tasks.rados.rados.0.trial153.stdout:797: oid 45 version 71 is already newer than 71 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:797: oid 45 version 71 is already newer than 71 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 71 (ObjNum 396 snap 95 seq_num 396) dirty exists 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:797: left oid 45 (ObjNum 396 snap 95 seq_num 396) 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:794: finishing write tid 2 to trial15342161-5 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:794: finishing write tid 3 to trial15342161-5 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:794: finishing write tid 4 to trial15342161-5 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:794: oid 5 updating version 0 to 78 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:794: oid 5 version 78 is already newer than 78 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:794: oid 5 version 78 is already newer than 76 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 5 v 78 (ObjNum 395 snap 95 seq_num 395) dirty exists 2026-02-20T12:02:35.714 INFO:tasks.rados.rados.0.trial153.stdout:794: left oid 5 (ObjNum 395 snap 95 seq_num 395) 2026-02-20T12:02:36.522 INFO:tasks.rados.rados.0.trial153.stdout:793: done (6 left) 2026-02-20T12:02:36.522 INFO:tasks.rados.rados.0.trial153.stdout:794: done (5 left) 2026-02-20T12:02:36.522 INFO:tasks.rados.rados.0.trial153.stdout:795: done (4 left) 2026-02-20T12:02:36.522 INFO:tasks.rados.rados.0.trial153.stdout:796: done (3 left) 2026-02-20T12:02:36.522 INFO:tasks.rados.rados.0.trial153.stdout:797: done (2 left) 2026-02-20T12:02:36.522 INFO:tasks.rados.rados.0.trial153.stdout:798: done (1 left) 2026-02-20T12:02:36.522 INFO:tasks.rados.rados.0.trial153.stdout:800: done (0 left) 2026-02-20T12:02:36.522 INFO:tasks.rados.rados.0.trial153.stdout:801: read oid 14 snap -1 2026-02-20T12:02:36.522 INFO:tasks.rados.rados.0.trial153.stdout:801: expect deleted 2026-02-20T12:02:36.523 INFO:tasks.rados.rados.0.trial153.stdout:802: write oid 19 current snap is 96 2026-02-20T12:02:36.523 INFO:tasks.rados.rados.0.trial153.stdout:802: seq_num 397 ranges {672379=724209,1865116=608472} 2026-02-20T12:02:36.527 INFO:tasks.rados.rados.0.trial153.stdout:802: writing trial15342161-19 from 672379 to 1396588 tid 1 2026-02-20T12:02:36.531 INFO:tasks.rados.rados.0.trial153.stdout:802: writing trial15342161-19 from 1865116 to 2473588 tid 2 2026-02-20T12:02:36.532 INFO:tasks.rados.rados.0.trial153.stdout:803: read oid 2 snap 94 2026-02-20T12:02:36.532 INFO:tasks.rados.rados.0.trial153.stdout:803: expect (ObjNum 329 snap 80 seq_num 329) 2026-02-20T12:02:36.532 INFO:tasks.rados.rados.0.trial153.stdout:804: write (excl) oid 18 current snap is 96 2026-02-20T12:02:36.532 INFO:tasks.rados.rados.0.trial153.stdout:804: seq_num 398 ranges {689186=799458,1923598=752130} 2026-02-20T12:02:36.537 INFO:tasks.rados.rados.0.trial153.stdout:804: writing trial15342161-18 from 689186 to 1488644 tid 1 2026-02-20T12:02:36.542 INFO:tasks.rados.rados.0.trial153.stdout:804: writing trial15342161-18 from 1923598 to 2675728 tid 2 2026-02-20T12:02:36.542 INFO:tasks.rados.rados.0.trial153.stdout:805: snap_remove snap 91 2026-02-20T12:02:36.675 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:36.754 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:36.749+0000 7f95a7b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:36.757 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:36.753+0000 7f95a7b5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:36.784 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:36.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:37.456 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~204s 2026-02-20T12:02:37.523 INFO:tasks.rados.rados.0.trial153.stdout:805: done (4 left) 2026-02-20T12:02:37.523 INFO:tasks.rados.rados.0.trial153.stdout:806: read oid 20 snap 94 2026-02-20T12:02:37.523 INFO:tasks.rados.rados.0.trial153.stdout:806: expect (ObjNum 339 snap 80 seq_num 339) 2026-02-20T12:02:37.523 INFO:tasks.rados.rados.0.trial153.stdout:807: read oid 34 snap -1 2026-02-20T12:02:37.523 INFO:tasks.rados.rados.0.trial153.stdout:807: expect (ObjNum 266 snap 70 seq_num 266) 2026-02-20T12:02:37.523 INFO:tasks.rados.rados.0.trial153.stdout:808: write (excl) oid 8 current snap is 96 2026-02-20T12:02:37.523 INFO:tasks.rados.rados.0.trial153.stdout:808: seq_num 399 ranges {705993=474707,1582080=612141} 2026-02-20T12:02:37.526 INFO:tasks.rados.rados.0.trial153.stdout:808: writing trial15342161-8 from 705993 to 1180700 tid 1 2026-02-20T12:02:37.530 INFO:tasks.rados.rados.0.trial153.stdout:808: writing trial15342161-8 from 1582080 to 2194221 tid 2 2026-02-20T12:02:37.530 INFO:tasks.rados.rados.0.trial153.stdout:809: read oid 28 snap -1 2026-02-20T12:02:37.530 INFO:tasks.rados.rados.0.trial153.stdout:809: expect (ObjNum 366 snap 87 seq_num 366) 2026-02-20T12:02:37.530 INFO:tasks.rados.rados.0.trial153.stdout:810: write (excl) oid 50 current snap is 96 2026-02-20T12:02:37.530 INFO:tasks.rados.rados.0.trial153.stdout:810: seq_num 400 ranges {722800=549956,2040562=682238} 2026-02-20T12:02:37.536 INFO:tasks.rados.rados.0.trial153.stdout:810: writing trial15342161-50 from 722800 to 1272756 tid 1 2026-02-20T12:02:37.541 INFO:tasks.rados.rados.0.trial153.stdout:810: writing trial15342161-50 from 2040562 to 2722800 tid 2 2026-02-20T12:02:37.541 INFO:tasks.rados.rados.0.trial153.stdout:811: write oid 38 current snap is 96 2026-02-20T12:02:37.541 INFO:tasks.rados.rados.0.trial153.stdout:811: seq_num 401 ranges {739607=625205,1982691=499457} 2026-02-20T12:02:37.546 INFO:tasks.rados.rados.0.trial153.stdout:811: writing trial15342161-38 from 739607 to 1364812 tid 1 2026-02-20T12:02:37.551 INFO:tasks.rados.rados.0.trial153.stdout:811: writing trial15342161-38 from 1982691 to 2482148 tid 2 2026-02-20T12:02:37.551 INFO:tasks.rados.rados.0.trial153.stdout:812: snap_create 2026-02-20T12:02:37.551 INFO:tasks.rados.rados.0.trial153.stdout:803: expect (ObjNum 329 snap 80 seq_num 329) 2026-02-20T12:02:37.551 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:37.605 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1508174 2026-02-20T12:02:37.639 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:37.686 INFO:tasks.rados.rados.0.trial153.stdout:802: finishing write tid 1 to trial15342161-19 2026-02-20T12:02:37.686 INFO:tasks.rados.rados.0.trial153.stdout:802: finishing write tid 2 to trial15342161-19 2026-02-20T12:02:37.686 INFO:tasks.rados.rados.0.trial153.stdout:804: finishing write tid 1 to trial15342161-18 2026-02-20T12:02:37.686 INFO:tasks.rados.rados.0.trial153.stdout:802: finishing write tid 3 to trial15342161-19 2026-02-20T12:02:37.686 INFO:tasks.rados.rados.0.trial153.stdout:802: finishing write tid 4 to trial15342161-19 2026-02-20T12:02:37.686 INFO:tasks.rados.rados.0.trial153.stdout:802: oid 19 updating version 0 to 208 2026-02-20T12:02:37.686 INFO:tasks.rados.rados.0.trial153.stdout:802: oid 19 version 208 is already newer than 208 2026-02-20T12:02:37.686 INFO:tasks.rados.rados.0.trial153.stdout:802: oid 19 version 208 is already newer than 208 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 208 (ObjNum 397 snap 96 seq_num 397) dirty exists 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:802: left oid 19 (ObjNum 397 snap 96 seq_num 397) 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:804: finishing write tid 2 to trial15342161-18 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:804: finishing write tid 3 to trial15342161-18 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:804: finishing write tid 4 to trial15342161-18 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:804: oid 18 updating version 0 to 61 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:804: oid 18 version 61 is already newer than 61 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:804: oid 18 version 61 is already newer than 59 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 18 v 61 (ObjNum 398 snap 96 seq_num 398) dirty exists 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:804: left oid 18 (ObjNum 398 snap 96 seq_num 398) 2026-02-20T12:02:37.687 INFO:tasks.rados.rados.0.trial153.stdout:809: expect (ObjNum 366 snap 87 seq_num 366) 2026-02-20T12:02:37.690 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1178763 2026-02-20T12:02:37.733 INFO:tasks.rados.rados.0.trial153.stdout:810: finishing write tid 1 to trial15342161-50 2026-02-20T12:02:37.734 INFO:tasks.rados.rados.0.trial153.stdout:807: expect (ObjNum 266 snap 70 seq_num 266) 2026-02-20T12:02:37.737 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1097240 2026-02-20T12:02:37.754 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1214888 2026-02-20T12:02:37.776 INFO:tasks.rados.rados.0.trial153.stdout:806: expect (ObjNum 339 snap 80 seq_num 339) 2026-02-20T12:02:37.776 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 185730 2026-02-20T12:02:37.839 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 764878 2026-02-20T12:02:37.958 INFO:tasks.rados.rados.0.trial153.stdout:811: finishing write tid 1 to trial15342161-38 2026-02-20T12:02:37.958 INFO:tasks.rados.rados.0.trial153.stdout:810: finishing write tid 2 to trial15342161-50 2026-02-20T12:02:37.958 INFO:tasks.rados.rados.0.trial153.stdout:811: finishing write tid 2 to trial15342161-38 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:810: finishing write tid 3 to trial15342161-50 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:811: finishing write tid 3 to trial15342161-38 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:810: finishing write tid 4 to trial15342161-50 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:810: oid 50 updating version 0 to 211 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:810: oid 50 version 211 is already newer than 209 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:810: oid 50 version 211 is already newer than 211 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 50 v 211 (ObjNum 400 snap 96 seq_num 400) dirty exists 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:810: left oid 50 (ObjNum 400 snap 96 seq_num 400) 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:811: finishing write tid 4 to trial15342161-38 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:811: oid 38 updating version 0 to 118 2026-02-20T12:02:37.959 INFO:tasks.rados.rados.0.trial153.stdout:811: oid 38 version 118 is already newer than 118 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:811: oid 38 version 118 is already newer than 118 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 38 v 118 (ObjNum 401 snap 96 seq_num 401) dirty exists 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:811: left oid 38 (ObjNum 401 snap 96 seq_num 401) 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:808: finishing write tid 1 to trial15342161-8 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:808: finishing write tid 2 to trial15342161-8 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:808: finishing write tid 3 to trial15342161-8 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:808: finishing write tid 4 to trial15342161-8 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:808: oid 8 updating version 0 to 82 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:808: oid 8 version 82 is already newer than 82 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:808: oid 8 version 82 is already newer than 80 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 8 v 82 (ObjNum 399 snap 96 seq_num 399) dirty exists 2026-02-20T12:02:37.960 INFO:tasks.rados.rados.0.trial153.stdout:808: left oid 8 (ObjNum 399 snap 96 seq_num 399) 2026-02-20T12:02:38.523 INFO:tasks.rados.rados.0.trial153.stdout:801: done (10 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:802: done (9 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:803: done (8 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:804: done (7 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:806: done (6 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:807: done (5 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:808: done (4 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:809: done (3 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:810: done (2 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:811: done (1 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:812: done (0 left) 2026-02-20T12:02:38.524 INFO:tasks.rados.rados.0.trial153.stdout:813: write (excl) oid 28 current snap is 97 2026-02-20T12:02:38.525 INFO:tasks.rados.rados.0.trial153.stdout:813: seq_num 402 ranges {756414=700454,2041173=715241} 2026-02-20T12:02:38.528 INFO:tasks.rados.rados.0.trial153.stdout:813: writing trial15342161-28 from 756414 to 1456868 tid 1 2026-02-20T12:02:38.534 INFO:tasks.rados.rados.0.trial153.stdout:813: writing trial15342161-28 from 2041173 to 2756414 tid 2 2026-02-20T12:02:38.534 INFO:tasks.rados.rados.0.trial153.stdout:814: write oid 14 current snap is 97 2026-02-20T12:02:38.534 INFO:tasks.rados.rados.0.trial153.stdout:814: seq_num 403 ranges {773221=492056,1816008=503126} 2026-02-20T12:02:38.538 INFO:tasks.rados.rados.0.trial153.stdout:814: writing trial15342161-14 from 773221 to 1265277 tid 1 2026-02-20T12:02:38.542 INFO:tasks.rados.rados.0.trial153.stdout:814: writing trial15342161-14 from 1816008 to 2319134 tid 2 2026-02-20T12:02:38.543 INFO:tasks.rados.rados.0.trial153.stdout:815: write oid 48 current snap is 97 2026-02-20T12:02:38.543 INFO:tasks.rados.rados.0.trial153.stdout:815: seq_num 404 ranges {790028=567305,1874490=763137} 2026-02-20T12:02:38.548 INFO:tasks.rados.rados.0.trial153.stdout:815: writing trial15342161-48 from 790028 to 1357333 tid 1 2026-02-20T12:02:38.554 INFO:tasks.rados.rados.0.trial153.stdout:815: writing trial15342161-48 from 1874490 to 2637627 tid 2 2026-02-20T12:02:38.554 INFO:tasks.rados.rados.0.trial153.stdout:816: write oid 42 current snap is 97 2026-02-20T12:02:38.554 INFO:tasks.rados.rados.0.trial153.stdout:816: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2026-02-20T12:02:38.558 INFO:tasks.rados.rados.0.trial153.stdout:816: writing trial15342161-42 from 406835 to 1049389 tid 1 2026-02-20T12:02:38.562 INFO:tasks.rados.rados.0.trial153.stdout:816: writing trial15342161-42 from 1532972 to 2039767 tid 2 2026-02-20T12:02:38.565 INFO:tasks.rados.rados.0.trial153.stdout:816: writing trial15342161-42 from 2772312 to 2806835 tid 3 2026-02-20T12:02:38.565 INFO:tasks.rados.rados.0.trial153.stdout:817: snap_remove snap 96 2026-02-20T12:02:38.802 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:38.877 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:38.871+0000 7f778f6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:38.879 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:38.875+0000 7f778f6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:38.905 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:38.921 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:39.525 INFO:tasks.rados.rados.0.trial153.stdout:817: done (4 left) 2026-02-20T12:02:39.526 INFO:tasks.rados.rados.0.trial153.stdout:818: write oid 47 current snap is 97 2026-02-20T12:02:39.526 INFO:tasks.rados.rados.0.trial153.stdout:814: finishing write tid 1 to trial15342161-14 2026-02-20T12:02:39.526 INFO:tasks.rados.rados.0.trial153.stdout:818: seq_num 406 ranges {423642=717803,1875101=766806} 2026-02-20T12:02:39.529 INFO:tasks.rados.rados.0.trial153.stdout:818: writing trial15342161-47 from 423642 to 1141445 tid 1 2026-02-20T12:02:39.535 INFO:tasks.rados.rados.0.trial153.stdout:818: writing trial15342161-47 from 1875101 to 2641907 tid 2 2026-02-20T12:02:39.535 INFO:tasks.rados.rados.0.trial153.stdout:819: write (excl) oid 39 current snap is 97 2026-02-20T12:02:39.535 INFO:tasks.rados.rados.0.trial153.stdout:815: finishing write tid 1 to trial15342161-48 2026-02-20T12:02:39.535 INFO:tasks.rados.rados.0.trial153.stdout:814: finishing write tid 2 to trial15342161-14 2026-02-20T12:02:39.536 INFO:tasks.rados.rados.0.trial153.stdout:819: seq_num 407 ranges {440449=793052,1933583=510464} 2026-02-20T12:02:39.539 INFO:tasks.rados.rados.0.trial153.stdout:819: writing trial15342161-39 from 440449 to 1233501 tid 1 2026-02-20T12:02:39.545 INFO:tasks.rados.rados.0.trial153.stdout:819: writing trial15342161-39 from 1933583 to 2444047 tid 2 2026-02-20T12:02:39.545 INFO:tasks.rados.rados.0.trial153.stdout:820: read oid 49 snap -1 2026-02-20T12:02:39.545 INFO:tasks.rados.rados.0.trial153.stdout:820: expect (ObjNum 357 snap 85 seq_num 357) 2026-02-20T12:02:39.545 INFO:tasks.rados.rados.0.trial153.stdout:821: read oid 1 snap -1 2026-02-20T12:02:39.545 INFO:tasks.rados.rados.0.trial153.stdout:821: expect deleted 2026-02-20T12:02:39.545 INFO:tasks.rados.rados.0.trial153.stdout:822: write oid 22 current snap is 97 2026-02-20T12:02:39.545 INFO:tasks.rados.rados.0.trial153.stdout:815: finishing write tid 2 to trial15342161-48 2026-02-20T12:02:39.545 INFO:tasks.rados.rados.0.trial153.stdout:816: finishing write tid 1 to trial15342161-42 2026-02-20T12:02:39.545 INFO:tasks.rados.rados.0.trial153.stdout:815: finishing write tid 3 to trial15342161-48 2026-02-20T12:02:39.545 INFO:tasks.rados.rados.0.trial153.stdout:822: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2026-02-20T12:02:39.548 INFO:tasks.rados.rados.0.trial153.stdout:822: writing trial15342161-22 from 457256 to 925557 tid 1 2026-02-20T12:02:39.553 INFO:tasks.rados.rados.0.trial153.stdout:822: writing trial15342161-22 from 1592065 to 2362540 tid 2 2026-02-20T12:02:39.555 INFO:tasks.rados.rados.0.trial153.stdout:822: writing trial15342161-22 from 2854581 to 2857256 tid 3 2026-02-20T12:02:39.555 INFO:tasks.rados.rados.0.trial153.stdout:823: snap_remove snap 95 2026-02-20T12:02:40.431 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:40.503 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:40.506 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:40.531 INFO:tasks.rados.rados.0.trial153.stdout:823: done (9 left) 2026-02-20T12:02:40.531 INFO:tasks.rados.rados.0.trial153.stdout:824: read oid 34 snap 93 2026-02-20T12:02:40.532 INFO:tasks.rados.rados.0.trial153.stdout:824: expect (ObjNum 266 snap 70 seq_num 266) 2026-02-20T12:02:40.532 INFO:tasks.rados.rados.0.trial153.stdout:825: delete oid 15 current snap is 97 2026-02-20T12:02:40.532 INFO:tasks.rados.rados.0.trial153.stdout:815: finishing write tid 4 to trial15342161-48 2026-02-20T12:02:40.532 INFO:tasks.rados.rados.0.trial153.stdout:815: oid 48 updating version 0 to 70 2026-02-20T12:02:40.532 INFO:tasks.rados.rados.0.trial153.stdout:815: oid 48 version 70 is already newer than 70 2026-02-20T12:02:40.532 INFO:tasks.rados.rados.0.trial153.stdout:815: oid 48 version 70 is already newer than 70 2026-02-20T12:02:40.532 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 48 v 70 (ObjNum 404 snap 97 seq_num 404) dirty exists 2026-02-20T12:02:40.532 INFO:tasks.rados.rados.0.trial153.stdout:815: left oid 48 (ObjNum 404 snap 97 seq_num 404) 2026-02-20T12:02:40.532 INFO:tasks.rados.rados.0.trial153.stdout:814: finishing write tid 3 to trial15342161-14 2026-02-20T12:02:40.532 INFO:tasks.rados.rados.0.trial153.stdout:816: finishing write tid 2 to trial15342161-42 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:814: finishing write tid 4 to trial15342161-14 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:814: oid 14 updating version 0 to 218 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:814: oid 14 version 218 is already newer than 218 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:814: oid 14 version 218 is already newer than 218 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 218 (ObjNum 403 snap 97 seq_num 403) dirty exists 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:814: left oid 14 (ObjNum 403 snap 97 seq_num 403) 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:816: finishing write tid 3 to trial15342161-42 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:816: finishing write tid 4 to trial15342161-42 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:816: finishing write tid 5 to trial15342161-42 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:816: oid 42 updating version 0 to 65 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:816: oid 42 version 65 is already newer than 65 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:816: oid 42 version 65 is already newer than 65 2026-02-20T12:02:40.533 INFO:tasks.rados.rados.0.trial153.stdout:816: oid 42 version 65 is already newer than 65 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 65 (ObjNum 405 snap 97 seq_num 405) dirty exists 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:816: left oid 42 (ObjNum 405 snap 97 seq_num 405) 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:813: finishing write tid 1 to trial15342161-28 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:813: finishing write tid 2 to trial15342161-28 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:813: finishing write tid 3 to trial15342161-28 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:813: finishing write tid 4 to trial15342161-28 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:813: oid 28 updating version 0 to 156 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:813: oid 28 version 156 is already newer than 156 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:813: oid 28 version 156 is already newer than 154 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 28 v 156 (ObjNum 402 snap 97 seq_num 402) dirty exists 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:813: left oid 28 (ObjNum 402 snap 97 seq_num 402) 2026-02-20T12:02:40.534 INFO:tasks.rados.rados.0.trial153.stdout:819: finishing write tid 1 to trial15342161-39 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:818: finishing write tid 1 to trial15342161-47 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:819: finishing write tid 2 to trial15342161-39 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:818: finishing write tid 2 to trial15342161-47 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:819: finishing write tid 3 to trial15342161-39 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:819: finishing write tid 4 to trial15342161-39 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:819: oid 39 updating version 0 to 177 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:819: oid 39 updating version 177 to 179 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:819: oid 39 version 179 is already newer than 179 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 39 v 179 (ObjNum 407 snap 97 seq_num 407) dirty exists 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:819: left oid 39 (ObjNum 407 snap 97 seq_num 407) 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:818: finishing write tid 3 to trial15342161-47 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:818: finishing write tid 4 to trial15342161-47 2026-02-20T12:02:40.535 INFO:tasks.rados.rados.0.trial153.stdout:818: oid 47 updating version 0 to 160 2026-02-20T12:02:40.536 INFO:tasks.rados.rados.0.trial153.stdout:818: oid 47 version 160 is already newer than 160 2026-02-20T12:02:40.536 INFO:tasks.rados.rados.0.trial153.stdout:818: oid 47 version 160 is already newer than 160 2026-02-20T12:02:40.536 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 47 v 160 (ObjNum 406 snap 97 seq_num 406) dirty exists 2026-02-20T12:02:40.536 INFO:tasks.rados.rados.0.trial153.stdout:818: left oid 47 (ObjNum 406 snap 97 seq_num 406) 2026-02-20T12:02:40.536 INFO:tasks.rados.rados.0.trial153.stdout:820: expect (ObjNum 357 snap 85 seq_num 357) 2026-02-20T12:02:40.628 INFO:tasks.rados.rados.0.trial153.stdout:822: finishing write tid 1 to trial15342161-22 2026-02-20T12:02:40.628 INFO:tasks.rados.rados.0.trial153.stdout:822: finishing write tid 2 to trial15342161-22 2026-02-20T12:02:40.628 INFO:tasks.rados.rados.0.trial153.stdout:822: finishing write tid 3 to trial15342161-22 2026-02-20T12:02:40.628 INFO:tasks.rados.rados.0.trial153.stdout:822: finishing write tid 4 to trial15342161-22 2026-02-20T12:02:40.628 INFO:tasks.rados.rados.0.trial153.stdout:813: done (10 left) 2026-02-20T12:02:40.628 INFO:tasks.rados.rados.0.trial153.stdout:814: done (9 left) 2026-02-20T12:02:40.628 INFO:tasks.rados.rados.0.trial153.stdout:815: done (8 left) 2026-02-20T12:02:40.628 INFO:tasks.rados.rados.0.trial153.stdout:816: done (7 left) 2026-02-20T12:02:40.628 INFO:tasks.rados.rados.0.trial153.stdout:818: done (6 left) 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:819: done (5 left) 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:820: done (4 left) 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:821: done (3 left) 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:825: done (2 left) 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:826: read oid 9 snap -1 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:826: expect (ObjNum 260 snap 68 seq_num 260) 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:827: read oid 12 snap -1 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:827: expect (ObjNum 358 snap 85 seq_num 358) 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:828: delete oid 32 current snap is 97 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:822: finishing write tid 5 to trial15342161-22 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:822: oid 22 updating version 0 to 165 2026-02-20T12:02:40.629 INFO:tasks.rados.rados.0.trial153.stdout:822: oid 22 version 165 is already newer than 165 2026-02-20T12:02:40.630 INFO:tasks.rados.rados.0.trial153.stdout:822: oid 22 version 165 is already newer than 165 2026-02-20T12:02:40.630 INFO:tasks.rados.rados.0.trial153.stdout:822: oid 22 version 165 is already newer than 165 2026-02-20T12:02:40.630 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 22 v 165 (ObjNum 408 snap 97 seq_num 408) dirty exists 2026-02-20T12:02:40.630 INFO:tasks.rados.rados.0.trial153.stdout:822: left oid 22 (ObjNum 408 snap 97 seq_num 408) 2026-02-20T12:02:40.630 INFO:tasks.rados.rados.0.trial153.stdout:824: expect (ObjNum 266 snap 70 seq_num 266) 2026-02-20T12:02:40.650 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 697788 2026-02-20T12:02:40.673 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2084224 2026-02-20T12:02:40.676 INFO:tasks.rados.rados.0.trial153.stdout:822: done (4 left) 2026-02-20T12:02:40.677 INFO:tasks.rados.rados.0.trial153.stdout:824: done (3 left) 2026-02-20T12:02:40.677 INFO:tasks.rados.rados.0.trial153.stdout:828: done (2 left) 2026-02-20T12:02:40.677 INFO:tasks.rados.rados.0.trial153.stdout:829: write (excl) oid 20 current snap is 97 2026-02-20T12:02:40.677 INFO:tasks.rados.rados.0.trial153.stdout:827: expect (ObjNum 358 snap 85 seq_num 358) 2026-02-20T12:02:40.677 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:40.710 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:40.728 INFO:tasks.rados.rados.0.trial153.stdout:826: expect (ObjNum 260 snap 68 seq_num 260) 2026-02-20T12:02:40.878 INFO:tasks.rados.rados.0.trial153.stdout:829: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2026-02-20T12:02:40.881 INFO:tasks.rados.rados.0.trial153.stdout:829: writing trial15342161-20 from 507677 to 918078 tid 1 2026-02-20T12:02:40.885 INFO:tasks.rados.rados.0.trial153.stdout:829: writing trial15342161-20 from 1367511 to 1885313 tid 2 2026-02-20T12:02:40.889 INFO:tasks.rados.rados.0.trial153.stdout:829: writing trial15342161-20 from 2420497 to 2907677 tid 3 2026-02-20T12:02:40.889 INFO:tasks.rados.rados.0.trial153.stdout:826: done (2 left) 2026-02-20T12:02:40.889 INFO:tasks.rados.rados.0.trial153.stdout:827: done (1 left) 2026-02-20T12:02:40.889 INFO:tasks.rados.rados.0.trial153.stdout:830: write (excl) oid 19 current snap is 97 2026-02-20T12:02:40.890 INFO:tasks.rados.rados.0.trial153.stdout:830: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2026-02-20T12:02:40.892 INFO:tasks.rados.rados.0.trial153.stdout:830: writing trial15342161-19 from 524484 to 1010134 tid 1 2026-02-20T12:02:40.897 INFO:tasks.rados.rados.0.trial153.stdout:830: writing trial15342161-19 from 1425993 to 2087453 tid 2 2026-02-20T12:02:40.901 INFO:tasks.rados.rados.0.trial153.stdout:830: writing trial15342161-19 from 2847920 to 2924484 tid 3 2026-02-20T12:02:40.901 INFO:tasks.rados.rados.0.trial153.stdout:831: read oid 47 snap -1 2026-02-20T12:02:40.901 INFO:tasks.rados.rados.0.trial153.stdout:831: expect (ObjNum 406 snap 97 seq_num 406) 2026-02-20T12:02:40.901 INFO:tasks.rados.rados.0.trial153.stdout:832: read oid 25 snap -1 2026-02-20T12:02:40.901 INFO:tasks.rados.rados.0.trial153.stdout:832: expect (ObjNum 389 snap 93 seq_num 389) 2026-02-20T12:02:40.901 INFO:tasks.rados.rados.0.trial153.stdout:833: write oid 35 current snap is 97 2026-02-20T12:02:40.901 INFO:tasks.rados.rados.0.trial153.stdout:833: seq_num 413 ranges {541291=560899,1884475=521471} 2026-02-20T12:02:40.904 INFO:tasks.rados.rados.0.trial153.stdout:833: writing trial15342161-35 from 541291 to 1102190 tid 1 2026-02-20T12:02:40.909 INFO:tasks.rados.rados.0.trial153.stdout:833: writing trial15342161-35 from 1884475 to 2405946 tid 2 2026-02-20T12:02:40.909 INFO:tasks.rados.rados.0.trial153.stdout:834: write oid 8 current snap is 97 2026-02-20T12:02:40.909 INFO:tasks.rados.rados.0.trial153.stdout:834: seq_num 414 ranges {558098=636148,1826604=665129} 2026-02-20T12:02:40.914 INFO:tasks.rados.rados.0.trial153.stdout:834: writing trial15342161-8 from 558098 to 1194246 tid 1 2026-02-20T12:02:40.919 INFO:tasks.rados.rados.0.trial153.stdout:834: writing trial15342161-8 from 1826604 to 2491733 tid 2 2026-02-20T12:02:40.919 INFO:tasks.rados.rados.0.trial153.stdout:835: write (excl) oid 42 current snap is 97 2026-02-20T12:02:40.920 INFO:tasks.rados.rados.0.trial153.stdout:830: finishing write tid 1 to trial15342161-19 2026-02-20T12:02:40.920 INFO:tasks.rados.rados.0.trial153.stdout:835: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2026-02-20T12:02:40.922 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:40.925 INFO:tasks.rados.rados.0.trial153.stdout:835: writing trial15342161-42 from 574905 to 1286302 tid 1 2026-02-20T12:02:40.928 INFO:tasks.rados.rados.0.trial153.stdout:835: writing trial15342161-42 from 1885086 to 2410226 tid 2 2026-02-20T12:02:40.931 INFO:tasks.rados.rados.0.trial153.stdout:835: writing trial15342161-42 from 2930189 to 2974905 tid 3 2026-02-20T12:02:40.931 INFO:tasks.rados.rados.0.trial153.stdout:836: snap_create 2026-02-20T12:02:40.942 INFO:tasks.rados.rados.0.trial153.stdout:830: finishing write tid 2 to trial15342161-19 2026-02-20T12:02:40.943 INFO:tasks.rados.rados.0.trial153.stdout:831: expect (ObjNum 406 snap 97 seq_num 406) 2026-02-20T12:02:40.943 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:40.992 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:41.005 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:40.999+0000 7fcd6cc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:41.007 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:41.003+0000 7fcd6cc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:41.034 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:41.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:41.089 INFO:tasks.rados.rados.0.trial153.stdout:835: finishing write tid 1 to trial15342161-42 2026-02-20T12:02:41.090 INFO:tasks.rados.rados.0.trial153.stdout:830: finishing write tid 3 to trial15342161-19 2026-02-20T12:02:41.090 INFO:tasks.rados.rados.0.trial153.stdout:829: finishing write tid 1 to trial15342161-20 2026-02-20T12:02:41.090 INFO:tasks.rados.rados.0.trial153.stdout:830: finishing write tid 4 to trial15342161-19 2026-02-20T12:02:41.090 INFO:tasks.rados.rados.0.trial153.stdout:833: finishing write tid 1 to trial15342161-35 2026-02-20T12:02:41.090 INFO:tasks.rados.rados.0.trial153.stdout:830: finishing write tid 5 to trial15342161-19 2026-02-20T12:02:41.090 INFO:tasks.rados.rados.0.trial153.stdout:830: oid 19 updating version 0 to 221 2026-02-20T12:02:41.090 INFO:tasks.rados.rados.0.trial153.stdout:830: oid 19 updating version 221 to 224 2026-02-20T12:02:41.090 INFO:tasks.rados.rados.0.trial153.stdout:830: oid 19 version 224 is already newer than 224 2026-02-20T12:02:41.090 INFO:tasks.rados.rados.0.trial153.stdout:830: oid 19 version 224 is already newer than 224 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 224 (ObjNum 412 snap 97 seq_num 412) dirty exists 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:830: left oid 19 (ObjNum 412 snap 97 seq_num 412) 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:835: finishing write tid 2 to trial15342161-42 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:834: finishing write tid 1 to trial15342161-8 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:833: finishing write tid 2 to trial15342161-35 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:835: finishing write tid 3 to trial15342161-42 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:834: finishing write tid 2 to trial15342161-8 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:829: finishing write tid 2 to trial15342161-20 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:833: finishing write tid 3 to trial15342161-35 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:834: finishing write tid 3 to trial15342161-8 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:834: finishing write tid 4 to trial15342161-8 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:834: oid 8 updating version 0 to 85 2026-02-20T12:02:41.091 INFO:tasks.rados.rados.0.trial153.stdout:834: oid 8 version 85 is already newer than 85 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:834: oid 8 version 85 is already newer than 85 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 8 v 85 (ObjNum 414 snap 97 seq_num 414) dirty exists 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:834: left oid 8 (ObjNum 414 snap 97 seq_num 414) 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:835: finishing write tid 4 to trial15342161-42 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:835: finishing write tid 5 to trial15342161-42 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:835: oid 42 updating version 0 to 69 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:835: oid 42 version 69 is already newer than 69 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:835: oid 42 version 69 is already newer than 66 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:835: oid 42 version 69 is already newer than 69 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 69 (ObjNum 415 snap 97 seq_num 415) dirty exists 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:835: left oid 42 (ObjNum 415 snap 97 seq_num 415) 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:833: finishing write tid 4 to trial15342161-35 2026-02-20T12:02:41.092 INFO:tasks.rados.rados.0.trial153.stdout:833: oid 35 updating version 0 to 169 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:833: oid 35 version 169 is already newer than 169 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:833: oid 35 version 169 is already newer than 169 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 35 v 169 (ObjNum 413 snap 97 seq_num 413) dirty exists 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:833: left oid 35 (ObjNum 413 snap 97 seq_num 413) 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:829: finishing write tid 3 to trial15342161-20 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:829: finishing write tid 4 to trial15342161-20 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:829: finishing write tid 5 to trial15342161-20 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:829: oid 20 updating version 0 to 81 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:829: oid 20 version 81 is already newer than 81 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:829: oid 20 version 81 is already newer than 81 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:829: oid 20 version 81 is already newer than 78 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 20 v 81 (ObjNum 411 snap 97 seq_num 411) dirty exists 2026-02-20T12:02:41.093 INFO:tasks.rados.rados.0.trial153.stdout:829: left oid 20 (ObjNum 411 snap 97 seq_num 411) 2026-02-20T12:02:41.094 INFO:tasks.rados.rados.0.trial153.stdout:832: expect (ObjNum 389 snap 93 seq_num 389) 2026-02-20T12:02:41.112 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2126859 2026-02-20T12:02:41.205 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2098027 2026-02-20T12:02:41.532 INFO:tasks.rados.rados.0.trial153.stdout:829: done (7 left) 2026-02-20T12:02:41.532 INFO:tasks.rados.rados.0.trial153.stdout:830: done (6 left) 2026-02-20T12:02:41.533 INFO:tasks.rados.rados.0.trial153.stdout:831: done (5 left) 2026-02-20T12:02:41.533 INFO:tasks.rados.rados.0.trial153.stdout:832: done (4 left) 2026-02-20T12:02:41.533 INFO:tasks.rados.rados.0.trial153.stdout:833: done (3 left) 2026-02-20T12:02:41.533 INFO:tasks.rados.rados.0.trial153.stdout:834: done (2 left) 2026-02-20T12:02:41.533 INFO:tasks.rados.rados.0.trial153.stdout:835: done (1 left) 2026-02-20T12:02:41.533 INFO:tasks.rados.rados.0.trial153.stdout:836: done (0 left) 2026-02-20T12:02:41.533 INFO:tasks.rados.rados.0.trial153.stdout:837: write oid 2 current snap is 98 2026-02-20T12:02:41.533 INFO:tasks.rados.rados.0.trial153.stdout:837: seq_num 416 ranges {591712=786646,1943568=668798} 2026-02-20T12:02:41.537 INFO:tasks.rados.rados.0.trial153.stdout:837: writing trial15342161-2 from 591712 to 1378358 tid 1 2026-02-20T12:02:41.543 INFO:tasks.rados.rados.0.trial153.stdout:837: writing trial15342161-2 from 1943568 to 2612366 tid 2 2026-02-20T12:02:41.543 INFO:tasks.rados.rados.0.trial153.stdout:838: read oid 6 snap -1 2026-02-20T12:02:41.543 INFO:tasks.rados.rados.0.trial153.stdout:838: expect deleted 2026-02-20T12:02:41.543 INFO:tasks.rados.rados.0.trial153.stdout:839: snap_remove snap 97 2026-02-20T12:02:42.536 INFO:tasks.rados.rados.0.trial153.stdout:839: done (2 left) 2026-02-20T12:02:42.536 INFO:tasks.rados.rados.0.trial153.stdout:840: snap_remove snap 94 2026-02-20T12:02:42.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~209s 2026-02-20T12:02:43.050 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:43.128 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:43.122+0000 7efeac635640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:43.130 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:43.126+0000 7efeac635640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:43.157 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:43.173 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:840: done (2 left) 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:841: snap_create 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:837: finishing write tid 1 to trial15342161-2 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:837: finishing write tid 2 to trial15342161-2 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:837: finishing write tid 3 to trial15342161-2 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:837: finishing write tid 4 to trial15342161-2 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:837: oid 2 updating version 0 to 253 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:837: oid 2 version 253 is already newer than 253 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:837: oid 2 version 253 is already newer than 253 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 2 v 253 (ObjNum 416 snap 98 seq_num 416) dirty exists 2026-02-20T12:02:43.541 INFO:tasks.rados.rados.0.trial153.stdout:837: left oid 2 (ObjNum 416 snap 98 seq_num 416) 2026-02-20T12:02:44.544 INFO:tasks.rados.rados.0.trial153.stdout:837: done (2 left) 2026-02-20T12:02:44.544 INFO:tasks.rados.rados.0.trial153.stdout:838: done (1 left) 2026-02-20T12:02:44.544 INFO:tasks.rados.rados.0.trial153.stdout:841: done (0 left) 2026-02-20T12:02:44.544 INFO:tasks.rados.rados.0.trial153.stdout:842: write (excl) oid 40 current snap is 99 2026-02-20T12:02:44.544 INFO:tasks.rados.rados.0.trial153.stdout:842: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2026-02-20T12:02:44.548 INFO:tasks.rados.rados.0.trial153.stdout:842: writing trial15342161-40 from 608519 to 1070414 tid 1 2026-02-20T12:02:44.552 INFO:tasks.rados.rados.0.trial153.stdout:842: writing trial15342161-40 from 1602050 to 2130859 tid 2 2026-02-20T12:02:44.556 INFO:tasks.rados.rados.0.trial153.stdout:842: writing trial15342161-40 from 2585035 to 3008519 tid 3 2026-02-20T12:02:44.556 INFO:tasks.rados.rados.0.trial153.stdout:843: write (excl) oid 48 current snap is 99 2026-02-20T12:02:44.556 INFO:tasks.rados.rados.0.trial153.stdout:843: seq_num 418 ranges {625326=537144,1944179=672467} 2026-02-20T12:02:44.560 INFO:tasks.rados.rados.0.trial153.stdout:843: writing trial15342161-48 from 625326 to 1162470 tid 1 2026-02-20T12:02:44.566 INFO:tasks.rados.rados.0.trial153.stdout:843: writing trial15342161-48 from 1944179 to 2616646 tid 2 2026-02-20T12:02:44.566 INFO:tasks.rados.rados.0.trial153.stdout:844: write (excl) oid 13 current snap is 99 2026-02-20T12:02:44.566 INFO:tasks.rados.rados.0.trial153.stdout:844: seq_num 419 ranges {642133=728746,2119014=532478} 2026-02-20T12:02:44.571 INFO:tasks.rados.rados.0.trial153.stdout:844: writing trial15342161-13 from 642133 to 1370879 tid 1 2026-02-20T12:02:44.576 INFO:tasks.rados.rados.0.trial153.stdout:844: writing trial15342161-13 from 2119014 to 2651492 tid 2 2026-02-20T12:02:44.576 INFO:tasks.rados.rados.0.trial153.stdout:845: write (excl) oid 43 current snap is 99 2026-02-20T12:02:44.576 INFO:tasks.rados.rados.0.trial153.stdout:845: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2026-02-20T12:02:44.580 INFO:tasks.rados.rados.0.trial153.stdout:845: writing trial15342161-43 from 658940 to 1062935 tid 1 2026-02-20T12:02:44.585 INFO:tasks.rados.rados.0.trial153.stdout:845: writing trial15342161-43 from 1777496 to 2453632 tid 2 2026-02-20T12:02:44.587 INFO:tasks.rados.rados.0.trial153.stdout:845: writing trial15342161-43 from 2950951 to 3058940 tid 3 2026-02-20T12:02:44.587 INFO:tasks.rados.rados.0.trial153.stdout:846: snap_remove snap 93 2026-02-20T12:02:45.174 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:45.253 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:45.248+0000 7f2432c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:45.256 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:45.252+0000 7f2432c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:45.283 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:45.299 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:45.507 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:45.547 INFO:tasks.rados.rados.0.trial153.stdout:846: done (4 left) 2026-02-20T12:02:45.547 INFO:tasks.rados.rados.0.trial153.stdout:847: write oid 3 current snap is 99 2026-02-20T12:02:45.547 INFO:tasks.rados.rados.0.trial153.stdout:842: finishing write tid 1 to trial15342161-40 2026-02-20T12:02:45.548 INFO:tasks.rados.rados.0.trial153.stdout:847: seq_num 421 ranges {675747=479244,1835978=536147} 2026-02-20T12:02:45.551 INFO:tasks.rados.rados.0.trial153.stdout:847: writing trial15342161-3 from 675747 to 1154991 tid 1 2026-02-20T12:02:45.558 INFO:tasks.rados.rados.0.trial153.stdout:847: writing trial15342161-3 from 1835978 to 2372125 tid 2 2026-02-20T12:02:45.558 INFO:tasks.rados.rados.0.trial153.stdout:848: snap_create 2026-02-20T12:02:45.558 INFO:tasks.rados.rados.0.trial153.stdout:842: finishing write tid 2 to trial15342161-40 2026-02-20T12:02:45.558 INFO:tasks.rados.rados.0.trial153.stdout:842: finishing write tid 3 to trial15342161-40 2026-02-20T12:02:45.558 INFO:tasks.rados.rados.0.trial153.stdout:844: finishing write tid 1 to trial15342161-13 2026-02-20T12:02:45.558 INFO:tasks.rados.rados.0.trial153.stdout:842: finishing write tid 4 to trial15342161-40 2026-02-20T12:02:45.558 INFO:tasks.rados.rados.0.trial153.stdout:842: finishing write tid 5 to trial15342161-40 2026-02-20T12:02:45.558 INFO:tasks.rados.rados.0.trial153.stdout:842: oid 40 updating version 0 to 173 2026-02-20T12:02:45.558 INFO:tasks.rados.rados.0.trial153.stdout:842: oid 40 version 173 is already newer than 170 2026-02-20T12:02:45.558 INFO:tasks.rados.rados.0.trial153.stdout:842: oid 40 version 173 is already newer than 173 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:842: oid 40 version 173 is already newer than 173 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 40 v 173 (ObjNum 417 snap 99 seq_num 417) dirty exists 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:842: left oid 40 (ObjNum 417 snap 99 seq_num 417) 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:845: finishing write tid 1 to trial15342161-43 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:844: finishing write tid 2 to trial15342161-13 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:845: finishing write tid 2 to trial15342161-43 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:845: finishing write tid 3 to trial15342161-43 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:844: finishing write tid 3 to trial15342161-13 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:844: finishing write tid 4 to trial15342161-13 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:844: oid 13 updating version 0 to 261 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:844: oid 13 version 261 is already newer than 261 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:844: oid 13 version 261 is already newer than 259 2026-02-20T12:02:45.559 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 13 v 261 (ObjNum 419 snap 99 seq_num 419) dirty exists 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:844: left oid 13 (ObjNum 419 snap 99 seq_num 419) 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:845: finishing write tid 4 to trial15342161-43 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:845: finishing write tid 5 to trial15342161-43 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:845: oid 43 updating version 0 to 76 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:845: oid 43 version 76 is already newer than 73 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:845: oid 43 version 76 is already newer than 76 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:845: oid 43 version 76 is already newer than 76 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 43 v 76 (ObjNum 420 snap 99 seq_num 420) dirty exists 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:845: left oid 43 (ObjNum 420 snap 99 seq_num 420) 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:843: finishing write tid 1 to trial15342161-48 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:843: finishing write tid 2 to trial15342161-48 2026-02-20T12:02:45.560 INFO:tasks.rados.rados.0.trial153.stdout:843: finishing write tid 3 to trial15342161-48 2026-02-20T12:02:45.561 INFO:tasks.rados.rados.0.trial153.stdout:843: finishing write tid 4 to trial15342161-48 2026-02-20T12:02:45.561 INFO:tasks.rados.rados.0.trial153.stdout:843: oid 48 updating version 0 to 78 2026-02-20T12:02:45.561 INFO:tasks.rados.rados.0.trial153.stdout:843: oid 48 updating version 78 to 80 2026-02-20T12:02:45.561 INFO:tasks.rados.rados.0.trial153.stdout:843: oid 48 version 80 is already newer than 80 2026-02-20T12:02:45.561 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 48 v 80 (ObjNum 418 snap 99 seq_num 418) dirty exists 2026-02-20T12:02:45.561 INFO:tasks.rados.rados.0.trial153.stdout:843: left oid 48 (ObjNum 418 snap 99 seq_num 418) 2026-02-20T12:02:45.582 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:45.585 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:45.595 INFO:tasks.rados.rados.0.trial153.stdout:847: finishing write tid 1 to trial15342161-3 2026-02-20T12:02:45.604 INFO:tasks.rados.rados.0.trial153.stdout:847: finishing write tid 2 to trial15342161-3 2026-02-20T12:02:45.611 INFO:tasks.rados.rados.0.trial153.stdout:847: finishing write tid 3 to trial15342161-3 2026-02-20T12:02:45.613 INFO:tasks.rados.rados.0.trial153.stdout:847: finishing write tid 4 to trial15342161-3 2026-02-20T12:02:45.613 INFO:tasks.rados.rados.0.trial153.stdout:847: oid 3 updating version 0 to 124 2026-02-20T12:02:45.613 INFO:tasks.rados.rados.0.trial153.stdout:847: oid 3 version 124 is already newer than 124 2026-02-20T12:02:45.613 INFO:tasks.rados.rados.0.trial153.stdout:847: oid 3 version 124 is already newer than 124 2026-02-20T12:02:45.613 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 3 v 124 (ObjNum 421 snap 99 seq_num 421) dirty exists 2026-02-20T12:02:45.613 INFO:tasks.rados.rados.0.trial153.stdout:847: left oid 3 (ObjNum 421 snap 99 seq_num 421) 2026-02-20T12:02:46.523 INFO:tasks.rados.rados.0.trial153.stdout:842: done (5 left) 2026-02-20T12:02:46.523 INFO:tasks.rados.rados.0.trial153.stdout:843: done (4 left) 2026-02-20T12:02:46.523 INFO:tasks.rados.rados.0.trial153.stdout:844: done (3 left) 2026-02-20T12:02:46.523 INFO:tasks.rados.rados.0.trial153.stdout:845: done (2 left) 2026-02-20T12:02:46.523 INFO:tasks.rados.rados.0.trial153.stdout:847: done (1 left) 2026-02-20T12:02:46.523 INFO:tasks.rados.rados.0.trial153.stdout:848: done (0 left) 2026-02-20T12:02:46.523 INFO:tasks.rados.rados.0.trial153.stdout:849: snap_create 2026-02-20T12:02:47.301 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:47.380 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:47.375+0000 7fbbcffe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:47.383 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:47.379+0000 7fbbcffe4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:47.410 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:47.426 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:47.528 INFO:tasks.rados.rados.0.trial153.stdout:849: done (0 left) 2026-02-20T12:02:47.528 INFO:tasks.rados.rados.0.trial153.stdout:850: read oid 17 snap -1 2026-02-20T12:02:47.528 INFO:tasks.rados.rados.0.trial153.stdout:850: expect (ObjNum 394 snap 95 seq_num 394) 2026-02-20T12:02:47.528 INFO:tasks.rados.rados.0.trial153.stdout:851: read oid 16 snap 99 2026-02-20T12:02:47.528 INFO:tasks.rados.rados.0.trial153.stdout:851: expect (ObjNum 287 snap 72 seq_num 287) 2026-02-20T12:02:47.528 INFO:tasks.rados.rados.0.trial153.stdout:852: snap_create 2026-02-20T12:02:47.556 INFO:tasks.rados.rados.0.trial153.stdout:850: expect (ObjNum 394 snap 95 seq_num 394) 2026-02-20T12:02:47.579 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:47.601 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:47.623 INFO:tasks.rados.rados.0.trial153.stdout:851: expect (ObjNum 287 snap 72 seq_num 287) 2026-02-20T12:02:47.679 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1958931 2026-02-20T12:02:48.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~215s 2026-02-20T12:02:48.529 INFO:tasks.rados.rados.0.trial153.stdout:850: done (2 left) 2026-02-20T12:02:48.529 INFO:tasks.rados.rados.0.trial153.stdout:851: done (1 left) 2026-02-20T12:02:48.529 INFO:tasks.rados.rados.0.trial153.stdout:852: done (0 left) 2026-02-20T12:02:48.529 INFO:tasks.rados.rados.0.trial153.stdout:853: read oid 46 snap -1 2026-02-20T12:02:48.529 INFO:tasks.rados.rados.0.trial153.stdout:853: expect deleted 2026-02-20T12:02:48.529 INFO:tasks.rados.rados.0.trial153.stdout:854: snap_create 2026-02-20T12:02:49.427 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:49.504 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:49.498+0000 7fbb27e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:49.506 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:49.502+0000 7fbb27e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:49.528 INFO:tasks.rados.rados.0.trial153.stdout:853: done (1 left) 2026-02-20T12:02:49.528 INFO:tasks.rados.rados.0.trial153.stdout:854: done (0 left) 2026-02-20T12:02:49.528 INFO:tasks.rados.rados.0.trial153.stdout:855: write oid 48 current snap is 103 2026-02-20T12:02:49.528 INFO:tasks.rados.rados.0.trial153.stdout:855: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2026-02-20T12:02:49.532 INFO:tasks.rados.rados.0.trial153.stdout:855: writing trial15342161-48 from 692554 to 1247047 tid 1 2026-02-20T12:02:49.535 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:49.537 INFO:tasks.rados.rados.0.trial153.stdout:855: writing trial15342161-48 from 1778107 to 2457912 tid 2 2026-02-20T12:02:49.539 INFO:tasks.rados.rados.0.trial153.stdout:855: writing trial15342161-48 from 2889444 to 3092554 tid 3 2026-02-20T12:02:49.539 INFO:tasks.rados.rados.0.trial153.stdout:856: snap_remove snap 99 2026-02-20T12:02:49.554 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:49.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:50.535 INFO:tasks.rados.rados.0.trial153.stdout:856: done (1 left) 2026-02-20T12:02:50.535 INFO:tasks.rados.rados.0.trial153.stdout:857: read oid 12 snap 100 2026-02-20T12:02:50.535 INFO:tasks.rados.rados.0.trial153.stdout:857: expect (ObjNum 358 snap 85 seq_num 358) 2026-02-20T12:02:50.535 INFO:tasks.rados.rados.0.trial153.stdout:858: snap_remove snap 102 2026-02-20T12:02:50.587 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:50.658 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:50.661 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:50.661 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:51.538 INFO:tasks.rados.rados.0.trial153.stdout:858: done (2 left) 2026-02-20T12:02:51.539 INFO:tasks.rados.rados.0.trial153.stdout:859: write (excl) oid 49 current snap is 103 2026-02-20T12:02:51.539 INFO:tasks.rados.rados.0.trial153.stdout:855: finishing write tid 1 to trial15342161-48 2026-02-20T12:02:51.539 INFO:tasks.rados.rados.0.trial153.stdout:859: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2026-02-20T12:02:51.543 INFO:tasks.rados.rados.0.trial153.stdout:859: writing trial15342161-49 from 709361 to 1339103 tid 1 2026-02-20T12:02:51.547 INFO:tasks.rados.rados.0.trial153.stdout:859: writing trial15342161-49 from 1836589 to 2376405 tid 2 2026-02-20T12:02:51.551 INFO:tasks.rados.rados.0.trial153.stdout:859: writing trial15342161-49 from 3033220 to 3109361 tid 3 2026-02-20T12:02:51.551 INFO:tasks.rados.rados.0.trial153.stdout:860: read oid 28 snap -1 2026-02-20T12:02:51.551 INFO:tasks.rados.rados.0.trial153.stdout:860: expect (ObjNum 402 snap 97 seq_num 402) 2026-02-20T12:02:51.551 INFO:tasks.rados.rados.0.trial153.stdout:861: write oid 24 current snap is 103 2026-02-20T12:02:51.551 INFO:tasks.rados.rados.0.trial153.stdout:855: finishing write tid 2 to trial15342161-48 2026-02-20T12:02:51.552 INFO:tasks.rados.rados.0.trial153.stdout:855: finishing write tid 3 to trial15342161-48 2026-02-20T12:02:51.552 INFO:tasks.rados.rados.0.trial153.stdout:855: finishing write tid 4 to trial15342161-48 2026-02-20T12:02:51.552 INFO:tasks.rados.rados.0.trial153.stdout:855: finishing write tid 5 to trial15342161-48 2026-02-20T12:02:51.552 INFO:tasks.rados.rados.0.trial153.stdout:855: oid 48 updating version 0 to 87 2026-02-20T12:02:51.552 INFO:tasks.rados.rados.0.trial153.stdout:855: oid 48 version 87 is already newer than 87 2026-02-20T12:02:51.552 INFO:tasks.rados.rados.0.trial153.stdout:855: oid 48 version 87 is already newer than 87 2026-02-20T12:02:51.552 INFO:tasks.rados.rados.0.trial153.stdout:855: oid 48 version 87 is already newer than 87 2026-02-20T12:02:51.552 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 48 v 87 (ObjNum 422 snap 103 seq_num 422) dirty exists 2026-02-20T12:02:51.552 INFO:tasks.rados.rados.0.trial153.stdout:855: left oid 48 (ObjNum 422 snap 103 seq_num 422) 2026-02-20T12:02:51.552 INFO:tasks.rados.rados.0.trial153.stdout:861: seq_num 424 ranges {726168=704991,1895071=683474} 2026-02-20T12:02:51.556 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:51.557 INFO:tasks.rados.rados.0.trial153.stdout:861: writing trial15342161-24 from 726168 to 1431159 tid 1 2026-02-20T12:02:51.562 INFO:tasks.rados.rados.0.trial153.stdout:861: writing trial15342161-24 from 1895071 to 2578545 tid 2 2026-02-20T12:02:51.562 INFO:tasks.rados.rados.0.trial153.stdout:855: done (4 left) 2026-02-20T12:02:51.562 INFO:tasks.rados.rados.0.trial153.stdout:862: write oid 17 current snap is 103 2026-02-20T12:02:51.562 INFO:tasks.rados.rados.0.trial153.stdout:857: expect (ObjNum 358 snap 85 seq_num 358) 2026-02-20T12:02:51.605 INFO:tasks.rados.rados.0.trial153.stdout:862: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2026-02-20T12:02:51.611 INFO:tasks.rados.rados.0.trial153.stdout:862: writing trial15342161-17 from 742975 to 1523215 tid 1 2026-02-20T12:02:51.615 INFO:tasks.rados.rados.0.trial153.stdout:862: writing trial15342161-17 from 1953553 to 2380685 tid 2 2026-02-20T12:02:51.618 INFO:tasks.rados.rados.0.trial153.stdout:862: writing trial15342161-17 from 2971713 to 3142975 tid 3 2026-02-20T12:02:51.618 INFO:tasks.rados.rados.0.trial153.stdout:857: done (4 left) 2026-02-20T12:02:51.618 INFO:tasks.rados.rados.0.trial153.stdout:863: snap_create 2026-02-20T12:02:51.618 INFO:tasks.rados.rados.0.trial153.stdout:861: finishing write tid 1 to trial15342161-24 2026-02-20T12:02:51.619 INFO:tasks.rados.rados.0.trial153.stdout:859: finishing write tid 1 to trial15342161-49 2026-02-20T12:02:51.619 INFO:tasks.rados.rados.0.trial153.stdout:860: expect (ObjNum 402 snap 97 seq_num 402) 2026-02-20T12:02:51.636 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:51.631+0000 7fd105c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:51.639 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:51.634+0000 7fd105c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:51.682 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:51.698 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:51.699 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:51.707 INFO:tasks.rados.rados.0.trial153.stdout:861: finishing write tid 2 to trial15342161-24 2026-02-20T12:02:51.707 INFO:tasks.rados.rados.0.trial153.stdout:859: finishing write tid 2 to trial15342161-49 2026-02-20T12:02:51.707 INFO:tasks.rados.rados.0.trial153.stdout:861: finishing write tid 3 to trial15342161-24 2026-02-20T12:02:51.707 INFO:tasks.rados.rados.0.trial153.stdout:861: finishing write tid 4 to trial15342161-24 2026-02-20T12:02:51.707 INFO:tasks.rados.rados.0.trial153.stdout:861: oid 24 updating version 0 to 28 2026-02-20T12:02:51.707 INFO:tasks.rados.rados.0.trial153.stdout:861: oid 24 version 28 is already newer than 28 2026-02-20T12:02:51.707 INFO:tasks.rados.rados.0.trial153.stdout:861: oid 24 version 28 is already newer than 28 2026-02-20T12:02:51.707 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 24 v 28 (ObjNum 424 snap 103 seq_num 424) dirty exists 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:861: left oid 24 (ObjNum 424 snap 103 seq_num 424) 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:859: finishing write tid 3 to trial15342161-49 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:859: finishing write tid 4 to trial15342161-49 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:859: finishing write tid 5 to trial15342161-49 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:859: oid 49 updating version 0 to 184 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:859: oid 49 version 184 is already newer than 181 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:859: oid 49 version 184 is already newer than 184 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:859: oid 49 version 184 is already newer than 184 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 49 v 184 (ObjNum 423 snap 103 seq_num 423) dirty exists 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:859: left oid 49 (ObjNum 423 snap 103 seq_num 423) 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:862: finishing write tid 1 to trial15342161-17 2026-02-20T12:02:51.708 INFO:tasks.rados.rados.0.trial153.stdout:862: finishing write tid 2 to trial15342161-17 2026-02-20T12:02:51.709 INFO:tasks.rados.rados.0.trial153.stdout:862: finishing write tid 3 to trial15342161-17 2026-02-20T12:02:51.709 INFO:tasks.rados.rados.0.trial153.stdout:862: finishing write tid 4 to trial15342161-17 2026-02-20T12:02:51.709 INFO:tasks.rados.rados.0.trial153.stdout:862: finishing write tid 5 to trial15342161-17 2026-02-20T12:02:51.709 INFO:tasks.rados.rados.0.trial153.stdout:862: oid 17 updating version 0 to 189 2026-02-20T12:02:51.709 INFO:tasks.rados.rados.0.trial153.stdout:862: oid 17 version 189 is already newer than 189 2026-02-20T12:02:51.709 INFO:tasks.rados.rados.0.trial153.stdout:862: oid 17 version 189 is already newer than 189 2026-02-20T12:02:51.709 INFO:tasks.rados.rados.0.trial153.stdout:862: oid 17 version 189 is already newer than 189 2026-02-20T12:02:51.709 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 17 v 189 (ObjNum 425 snap 103 seq_num 425) dirty exists 2026-02-20T12:02:51.709 INFO:tasks.rados.rados.0.trial153.stdout:862: left oid 17 (ObjNum 425 snap 103 seq_num 425) 2026-02-20T12:02:52.541 INFO:tasks.rados.rados.0.trial153.stdout:859: done (4 left) 2026-02-20T12:02:52.541 INFO:tasks.rados.rados.0.trial153.stdout:860: done (3 left) 2026-02-20T12:02:52.541 INFO:tasks.rados.rados.0.trial153.stdout:861: done (2 left) 2026-02-20T12:02:52.542 INFO:tasks.rados.rados.0.trial153.stdout:862: done (1 left) 2026-02-20T12:02:52.542 INFO:tasks.rados.rados.0.trial153.stdout:863: done (0 left) 2026-02-20T12:02:52.542 INFO:tasks.rados.rados.0.trial153.stdout:864: write oid 34 current snap is 104 2026-02-20T12:02:52.542 INFO:tasks.rados.rados.0.trial153.stdout:864: seq_num 426 ranges {759782=571842,2012035=687143} 2026-02-20T12:02:52.546 INFO:tasks.rados.rados.0.trial153.stdout:864: writing trial15342161-34 from 759782 to 1331624 tid 1 2026-02-20T12:02:52.551 INFO:tasks.rados.rados.0.trial153.stdout:864: writing trial15342161-34 from 2012035 to 2699178 tid 2 2026-02-20T12:02:52.552 INFO:tasks.rados.rados.0.trial153.stdout:865: snap_create 2026-02-20T12:02:52.588 INFO:tasks.rados.rados.0.trial153.stdout:864: finishing write tid 1 to trial15342161-34 2026-02-20T12:02:52.598 INFO:tasks.rados.rados.0.trial153.stdout:864: finishing write tid 2 to trial15342161-34 2026-02-20T12:02:52.606 INFO:tasks.rados.rados.0.trial153.stdout:864: finishing write tid 3 to trial15342161-34 2026-02-20T12:02:52.608 INFO:tasks.rados.rados.0.trial153.stdout:864: finishing write tid 4 to trial15342161-34 2026-02-20T12:02:52.608 INFO:tasks.rados.rados.0.trial153.stdout:864: oid 34 updating version 0 to 139 2026-02-20T12:02:52.608 INFO:tasks.rados.rados.0.trial153.stdout:864: oid 34 version 139 is already newer than 139 2026-02-20T12:02:52.608 INFO:tasks.rados.rados.0.trial153.stdout:864: oid 34 version 139 is already newer than 139 2026-02-20T12:02:52.608 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 139 (ObjNum 426 snap 104 seq_num 426) dirty exists 2026-02-20T12:02:52.608 INFO:tasks.rados.rados.0.trial153.stdout:864: left oid 34 (ObjNum 426 snap 104 seq_num 426) 2026-02-20T12:02:53.547 INFO:tasks.rados.rados.0.trial153.stdout:864: done (1 left) 2026-02-20T12:02:53.547 INFO:tasks.rados.rados.0.trial153.stdout:865: done (0 left) 2026-02-20T12:02:53.547 INFO:tasks.rados.rados.0.trial153.stdout:866: snap_remove snap 104 2026-02-20T12:02:53.700 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:53.776 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:53.771+0000 7fe6961c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:53.779 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:53.774+0000 7fe6961c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:53.806 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:53.821 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:53.969 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~220s 2026-02-20T12:02:54.551 INFO:tasks.rados.rados.0.trial153.stdout:866: done (0 left) 2026-02-20T12:02:54.551 INFO:tasks.rados.rados.0.trial153.stdout:867: read oid 44 snap -1 2026-02-20T12:02:54.551 INFO:tasks.rados.rados.0.trial153.stdout:867: expect (ObjNum 392 snap 95 seq_num 392) 2026-02-20T12:02:54.551 INFO:tasks.rados.rados.0.trial153.stdout:868: read oid 23 snap -1 2026-02-20T12:02:54.552 INFO:tasks.rados.rados.0.trial153.stdout:868: expect (ObjNum 391 snap 94 seq_num 391) 2026-02-20T12:02:54.552 INFO:tasks.rados.rados.0.trial153.stdout:869: write (excl) oid 14 current snap is 105 2026-02-20T12:02:54.552 INFO:tasks.rados.rados.0.trial153.stdout:869: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2026-02-20T12:02:54.556 INFO:tasks.rados.rados.0.trial153.stdout:869: writing trial15342161-14 from 776589 to 1423680 tid 1 2026-02-20T12:02:54.560 INFO:tasks.rados.rados.0.trial153.stdout:869: writing trial15342161-14 from 2070517 to 2501318 tid 2 2026-02-20T12:02:54.563 INFO:tasks.rados.rados.0.trial153.stdout:869: writing trial15342161-14 from 3026559 to 3176589 tid 3 2026-02-20T12:02:54.563 INFO:tasks.rados.rados.0.trial153.stdout:870: snap_create 2026-02-20T12:02:54.581 INFO:tasks.rados.rados.0.trial153.stdout:868: expect (ObjNum 391 snap 94 seq_num 391) 2026-02-20T12:02:54.645 INFO:tasks.rados.rados.0.trial153.stdout:867: expect (ObjNum 392 snap 95 seq_num 392) 2026-02-20T12:02:54.696 INFO:tasks.rados.rados.0.trial153.stdout:869: finishing write tid 1 to trial15342161-14 2026-02-20T12:02:54.697 INFO:tasks.rados.rados.0.trial153.stdout:869: finishing write tid 2 to trial15342161-14 2026-02-20T12:02:54.697 INFO:tasks.rados.rados.0.trial153.stdout:869: finishing write tid 3 to trial15342161-14 2026-02-20T12:02:54.697 INFO:tasks.rados.rados.0.trial153.stdout:869: finishing write tid 4 to trial15342161-14 2026-02-20T12:02:54.697 INFO:tasks.rados.rados.0.trial153.stdout:869: finishing write tid 5 to trial15342161-14 2026-02-20T12:02:54.697 INFO:tasks.rados.rados.0.trial153.stdout:869: oid 14 updating version 0 to 234 2026-02-20T12:02:54.697 INFO:tasks.rados.rados.0.trial153.stdout:869: oid 14 updating version 234 to 237 2026-02-20T12:02:54.697 INFO:tasks.rados.rados.0.trial153.stdout:869: oid 14 version 237 is already newer than 237 2026-02-20T12:02:54.697 INFO:tasks.rados.rados.0.trial153.stdout:869: oid 14 version 237 is already newer than 237 2026-02-20T12:02:54.697 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 237 (ObjNum 427 snap 105 seq_num 427) dirty exists 2026-02-20T12:02:54.697 INFO:tasks.rados.rados.0.trial153.stdout:869: left oid 14 (ObjNum 427 snap 105 seq_num 427) 2026-02-20T12:02:55.555 INFO:tasks.rados.rados.0.trial153.stdout:867: done (3 left) 2026-02-20T12:02:55.555 INFO:tasks.rados.rados.0.trial153.stdout:868: done (2 left) 2026-02-20T12:02:55.555 INFO:tasks.rados.rados.0.trial153.stdout:869: done (1 left) 2026-02-20T12:02:55.555 INFO:tasks.rados.rados.0.trial153.stdout:870: done (0 left) 2026-02-20T12:02:55.555 INFO:tasks.rados.rados.0.trial153.stdout:871: read oid 12 snap -1 2026-02-20T12:02:55.555 INFO:tasks.rados.rados.0.trial153.stdout:871: expect (ObjNum 358 snap 85 seq_num 358) 2026-02-20T12:02:55.555 INFO:tasks.rados.rados.0.trial153.stdout:872: write (excl) oid 30 current snap is 106 2026-02-20T12:02:55.556 INFO:tasks.rados.rados.0.trial153.stdout:872: seq_num 428 ranges {793396=722340,2128999=690812} 2026-02-20T12:02:55.560 INFO:tasks.rados.rados.0.trial153.stdout:872: writing trial15342161-30 from 793396 to 1515736 tid 1 2026-02-20T12:02:55.565 INFO:tasks.rados.rados.0.trial153.stdout:872: writing trial15342161-30 from 2128999 to 2819811 tid 2 2026-02-20T12:02:55.566 INFO:tasks.rados.rados.0.trial153.stdout:873: write oid 20 current snap is 106 2026-02-20T12:02:55.566 INFO:tasks.rados.rados.0.trial153.stdout:873: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2026-02-20T12:02:55.570 INFO:tasks.rados.rados.0.trial153.stdout:873: writing trial15342161-20 from 410203 to 1207792 tid 1 2026-02-20T12:02:55.574 INFO:tasks.rados.rados.0.trial153.stdout:873: writing trial15342161-20 from 1787481 to 2221951 tid 2 2026-02-20T12:02:55.578 INFO:tasks.rados.rados.0.trial153.stdout:873: writing trial15342161-20 from 2681405 to 3210203 tid 3 2026-02-20T12:02:55.578 INFO:tasks.rados.rados.0.trial153.stdout:874: write oid 42 current snap is 106 2026-02-20T12:02:55.578 INFO:tasks.rados.rados.0.trial153.stdout:874: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2026-02-20T12:02:55.581 INFO:tasks.rados.rados.0.trial153.stdout:874: writing trial15342161-42 from 427010 to 899848 tid 1 2026-02-20T12:02:55.585 INFO:tasks.rados.rados.0.trial153.stdout:874: writing trial15342161-42 from 1329610 to 2024091 tid 2 2026-02-20T12:02:55.590 INFO:tasks.rados.rados.0.trial153.stdout:874: writing trial15342161-42 from 2708828 to 3227010 tid 3 2026-02-20T12:02:55.591 INFO:tasks.rados.rados.0.trial153.stdout:875: write oid 16 current snap is 106 2026-02-20T12:02:55.591 INFO:tasks.rados.rados.0.trial153.stdout:875: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2026-02-20T12:02:55.595 INFO:tasks.rados.rados.0.trial153.stdout:875: writing trial15342161-16 from 443817 to 991904 tid 1 2026-02-20T12:02:55.599 INFO:tasks.rados.rados.0.trial153.stdout:875: writing trial15342161-16 from 1788092 to 2226231 tid 2 2026-02-20T12:02:55.603 INFO:tasks.rados.rados.0.trial153.stdout:875: writing trial15342161-16 from 3019898 to 3243817 tid 3 2026-02-20T12:02:55.603 INFO:tasks.rados.rados.0.trial153.stdout:876: delete oid 34 current snap is 106 2026-02-20T12:02:55.603 INFO:tasks.rados.rados.0.trial153.stdout:871: expect (ObjNum 358 snap 85 seq_num 358) 2026-02-20T12:02:55.620 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:55.638 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:55.655 INFO:tasks.rados.rados.0.trial153.stdout:874: finishing write tid 1 to trial15342161-42 2026-02-20T12:02:55.655 INFO:tasks.rados.rados.0.trial153.stdout:872: finishing write tid 1 to trial15342161-30 2026-02-20T12:02:55.656 INFO:tasks.rados.rados.0.trial153.stdout:874: finishing write tid 2 to trial15342161-42 2026-02-20T12:02:55.656 INFO:tasks.rados.rados.0.trial153.stdout:874: finishing write tid 3 to trial15342161-42 2026-02-20T12:02:55.656 INFO:tasks.rados.rados.0.trial153.stdout:874: finishing write tid 4 to trial15342161-42 2026-02-20T12:02:55.656 INFO:tasks.rados.rados.0.trial153.stdout:874: finishing write tid 5 to trial15342161-42 2026-02-20T12:02:55.656 INFO:tasks.rados.rados.0.trial153.stdout:874: oid 42 updating version 0 to 74 2026-02-20T12:02:55.656 INFO:tasks.rados.rados.0.trial153.stdout:874: oid 42 version 74 is already newer than 74 2026-02-20T12:02:55.656 INFO:tasks.rados.rados.0.trial153.stdout:874: oid 42 version 74 is already newer than 74 2026-02-20T12:02:55.656 INFO:tasks.rados.rados.0.trial153.stdout:874: oid 42 version 74 is already newer than 74 2026-02-20T12:02:55.656 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 74 (ObjNum 430 snap 106 seq_num 430) dirty exists 2026-02-20T12:02:55.656 INFO:tasks.rados.rados.0.trial153.stdout:874: left oid 42 (ObjNum 430 snap 106 seq_num 430) 2026-02-20T12:02:55.660 INFO:tasks.rados.rados.0.trial153.stdout:872: finishing write tid 2 to trial15342161-30 2026-02-20T12:02:55.663 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:02:55.666 INFO:tasks.rados.rados.0.trial153.stdout:873: finishing write tid 1 to trial15342161-20 2026-02-20T12:02:55.677 INFO:tasks.rados.rados.0.trial153.stdout:873: finishing write tid 2 to trial15342161-20 2026-02-20T12:02:55.691 INFO:tasks.rados.rados.0.trial153.stdout:872: finishing write tid 3 to trial15342161-30 2026-02-20T12:02:55.694 INFO:tasks.rados.rados.0.trial153.stdout:873: finishing write tid 3 to trial15342161-20 2026-02-20T12:02:55.696 INFO:tasks.rados.rados.0.trial153.stdout:872: finishing write tid 4 to trial15342161-30 2026-02-20T12:02:55.696 INFO:tasks.rados.rados.0.trial153.stdout:872: oid 30 updating version 0 to 141 2026-02-20T12:02:55.696 INFO:tasks.rados.rados.0.trial153.stdout:872: oid 30 updating version 141 to 143 2026-02-20T12:02:55.696 INFO:tasks.rados.rados.0.trial153.stdout:872: oid 30 version 143 is already newer than 143 2026-02-20T12:02:55.696 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 30 v 143 (ObjNum 428 snap 106 seq_num 428) dirty exists 2026-02-20T12:02:55.696 INFO:tasks.rados.rados.0.trial153.stdout:872: left oid 30 (ObjNum 428 snap 106 seq_num 428) 2026-02-20T12:02:55.700 INFO:tasks.rados.rados.0.trial153.stdout:873: finishing write tid 4 to trial15342161-20 2026-02-20T12:02:55.704 INFO:tasks.rados.rados.0.trial153.stdout:873: finishing write tid 5 to trial15342161-20 2026-02-20T12:02:55.704 INFO:tasks.rados.rados.0.trial153.stdout:873: oid 20 updating version 0 to 86 2026-02-20T12:02:55.704 INFO:tasks.rados.rados.0.trial153.stdout:873: oid 20 version 86 is already newer than 86 2026-02-20T12:02:55.704 INFO:tasks.rados.rados.0.trial153.stdout:873: oid 20 version 86 is already newer than 86 2026-02-20T12:02:55.704 INFO:tasks.rados.rados.0.trial153.stdout:873: oid 20 version 86 is already newer than 86 2026-02-20T12:02:55.704 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 20 v 86 (ObjNum 429 snap 106 seq_num 429) dirty exists 2026-02-20T12:02:55.704 INFO:tasks.rados.rados.0.trial153.stdout:873: left oid 20 (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:02:55.710 INFO:tasks.rados.rados.0.trial153.stdout:875: finishing write tid 1 to trial15342161-16 2026-02-20T12:02:55.718 INFO:tasks.rados.rados.0.trial153.stdout:875: finishing write tid 2 to trial15342161-16 2026-02-20T12:02:55.723 INFO:tasks.rados.rados.0.trial153.stdout:875: finishing write tid 3 to trial15342161-16 2026-02-20T12:02:55.727 INFO:tasks.rados.rados.0.trial153.stdout:875: finishing write tid 4 to trial15342161-16 2026-02-20T12:02:55.729 INFO:tasks.rados.rados.0.trial153.stdout:875: finishing write tid 5 to trial15342161-16 2026-02-20T12:02:55.730 INFO:tasks.rados.rados.0.trial153.stdout:875: oid 16 updating version 0 to 148 2026-02-20T12:02:55.730 INFO:tasks.rados.rados.0.trial153.stdout:875: oid 16 version 148 is already newer than 148 2026-02-20T12:02:55.730 INFO:tasks.rados.rados.0.trial153.stdout:875: oid 16 version 148 is already newer than 148 2026-02-20T12:02:55.730 INFO:tasks.rados.rados.0.trial153.stdout:875: oid 16 version 148 is already newer than 148 2026-02-20T12:02:55.730 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 16 v 148 (ObjNum 431 snap 106 seq_num 431) dirty exists 2026-02-20T12:02:55.730 INFO:tasks.rados.rados.0.trial153.stdout:875: left oid 16 (ObjNum 431 snap 106 seq_num 431) 2026-02-20T12:02:55.734 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:02:55.737 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:02:55.737 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:02:55.744 INFO:tasks.rados.rados.0.trial153.stdout:871: done (5 left) 2026-02-20T12:02:55.744 INFO:tasks.rados.rados.0.trial153.stdout:872: done (4 left) 2026-02-20T12:02:55.744 INFO:tasks.rados.rados.0.trial153.stdout:873: done (3 left) 2026-02-20T12:02:55.744 INFO:tasks.rados.rados.0.trial153.stdout:874: done (2 left) 2026-02-20T12:02:55.744 INFO:tasks.rados.rados.0.trial153.stdout:875: done (1 left) 2026-02-20T12:02:55.744 INFO:tasks.rados.rados.0.trial153.stdout:876: done (0 left) 2026-02-20T12:02:55.744 INFO:tasks.rados.rados.0.trial153.stdout:877: snap_remove snap 103 2026-02-20T12:02:55.823 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:02:55.898 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:55.893+0000 7fbdd1fb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:55.900 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:55.896+0000 7fbdd1fb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:55.926 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:55.943 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:56.525 INFO:tasks.rados.rados.0.trial153.stdout:877: done (0 left) 2026-02-20T12:02:56.525 INFO:tasks.rados.rados.0.trial153.stdout:878: read oid 21 snap 100 2026-02-20T12:02:56.525 INFO:tasks.rados.rados.0.trial153.stdout:878: expect deleted 2026-02-20T12:02:56.525 INFO:tasks.rados.rados.0.trial153.stdout:879: read oid 2 snap -1 2026-02-20T12:02:56.525 INFO:tasks.rados.rados.0.trial153.stdout:879: expect (ObjNum 416 snap 98 seq_num 416) 2026-02-20T12:02:56.525 INFO:tasks.rados.rados.0.trial153.stdout:880: delete oid 40 current snap is 106 2026-02-20T12:02:56.571 INFO:tasks.rados.rados.0.trial153.stdout:879: expect (ObjNum 416 snap 98 seq_num 416) 2026-02-20T12:02:56.665 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:56.717 INFO:tasks.rados.rados.0.trial153.stdout:878: done (2 left) 2026-02-20T12:02:56.717 INFO:tasks.rados.rados.0.trial153.stdout:879: done (1 left) 2026-02-20T12:02:56.717 INFO:tasks.rados.rados.0.trial153.stdout:880: done (0 left) 2026-02-20T12:02:56.717 INFO:tasks.rados.rados.0.trial153.stdout:881: write oid 14 current snap is 106 2026-02-20T12:02:56.717 INFO:tasks.rados.rados.0.trial153.stdout:881: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2026-02-20T12:02:56.720 INFO:tasks.rados.rados.0.trial153.stdout:881: writing trial15342161-14 from 494238 to 984425 tid 1 2026-02-20T12:02:56.725 INFO:tasks.rados.rados.0.trial153.stdout:881: writing trial15342161-14 from 1563538 to 2265357 tid 2 2026-02-20T12:02:56.730 INFO:tasks.rados.rados.0.trial153.stdout:881: writing trial15342161-14 from 2818520 to 3294238 tid 3 2026-02-20T12:02:56.730 INFO:tasks.rados.rados.0.trial153.stdout:882: read oid 20 snap 101 2026-02-20T12:02:56.730 INFO:tasks.rados.rados.0.trial153.stdout:882: expect (ObjNum 411 snap 97 seq_num 411) 2026-02-20T12:02:56.730 INFO:tasks.rados.rados.0.trial153.stdout:883: read oid 11 snap -1 2026-02-20T12:02:56.730 INFO:tasks.rados.rados.0.trial153.stdout:883: expect (ObjNum 377 snap 89 seq_num 377) 2026-02-20T12:02:56.730 INFO:tasks.rados.rados.0.trial153.stdout:884: snap_create 2026-02-20T12:02:56.744 INFO:tasks.rados.rados.0.trial153.stdout:883: expect (ObjNum 377 snap 89 seq_num 377) 2026-02-20T12:02:56.747 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1494313 2026-02-20T12:02:56.767 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:56.786 INFO:tasks.rados.rados.0.trial153.stdout:882: expect (ObjNum 411 snap 97 seq_num 411) 2026-02-20T12:02:56.811 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2164359 2026-02-20T12:02:56.822 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:56.864 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:881: finishing write tid 1 to trial15342161-14 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:881: finishing write tid 2 to trial15342161-14 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:881: finishing write tid 3 to trial15342161-14 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:881: finishing write tid 4 to trial15342161-14 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:881: finishing write tid 5 to trial15342161-14 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:881: oid 14 updating version 0 to 244 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:881: oid 14 version 244 is already newer than 244 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:881: oid 14 version 244 is already newer than 244 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:881: oid 14 version 244 is already newer than 244 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 244 (ObjNum 434 snap 106 seq_num 434) dirty exists 2026-02-20T12:02:56.907 INFO:tasks.rados.rados.0.trial153.stdout:881: left oid 14 (ObjNum 434 snap 106 seq_num 434) 2026-02-20T12:02:57.560 INFO:tasks.rados.rados.0.trial153.stdout:881: done (3 left) 2026-02-20T12:02:57.560 INFO:tasks.rados.rados.0.trial153.stdout:882: done (2 left) 2026-02-20T12:02:57.560 INFO:tasks.rados.rados.0.trial153.stdout:883: done (1 left) 2026-02-20T12:02:57.560 INFO:tasks.rados.rados.0.trial153.stdout:884: done (0 left) 2026-02-20T12:02:57.560 INFO:tasks.rados.rados.0.trial153.stdout:885: write oid 44 current snap is 107 2026-02-20T12:02:57.560 INFO:tasks.rados.rados.0.trial153.stdout:885: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2026-02-20T12:02:57.564 INFO:tasks.rados.rados.0.trial153.stdout:885: writing trial15342161-44 from 511045 to 1076481 tid 1 2026-02-20T12:02:57.569 INFO:tasks.rados.rados.0.trial153.stdout:885: writing trial15342161-44 from 1622020 to 2067497 tid 2 2026-02-20T12:02:57.574 INFO:tasks.rados.rados.0.trial153.stdout:885: writing trial15342161-44 from 2729590 to 3286445 tid 3 2026-02-20T12:02:57.574 INFO:tasks.rados.rados.0.trial153.stdout:886: read oid 18 snap -1 2026-02-20T12:02:57.575 INFO:tasks.rados.rados.0.trial153.stdout:886: expect (ObjNum 398 snap 96 seq_num 398) 2026-02-20T12:02:57.575 INFO:tasks.rados.rados.0.trial153.stdout:887: read oid 19 snap -1 2026-02-20T12:02:57.575 INFO:tasks.rados.rados.0.trial153.stdout:887: expect (ObjNum 412 snap 97 seq_num 412) 2026-02-20T12:02:57.575 INFO:tasks.rados.rados.0.trial153.stdout:888: snap_remove snap 105 2026-02-20T12:02:57.943 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:02:58.022 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:58.016+0000 7efda245a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:58.024 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:02:58.020+0000 7efda245a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:02:58.051 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:02:58.067 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:02:58.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:02:58.563 INFO:tasks.rados.rados.0.trial153.stdout:888: done (3 left) 2026-02-20T12:02:58.563 INFO:tasks.rados.rados.0.trial153.stdout:889: write oid 45 current snap is 107 2026-02-20T12:02:58.563 INFO:tasks.rados.rados.0.trial153.stdout:889: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2026-02-20T12:02:58.567 INFO:tasks.rados.rados.0.trial153.stdout:889: writing trial15342161-45 from 527852 to 1168537 tid 1 2026-02-20T12:02:58.572 INFO:tasks.rados.rados.0.trial153.stdout:889: writing trial15342161-45 from 1680502 to 2269637 tid 2 2026-02-20T12:02:58.576 INFO:tasks.rados.rados.0.trial153.stdout:889: writing trial15342161-45 from 2757013 to 3327852 tid 3 2026-02-20T12:02:58.577 INFO:tasks.rados.rados.0.trial153.stdout:890: delete oid 29 current snap is 107 2026-02-20T12:02:58.577 INFO:tasks.rados.rados.0.trial153.stdout:886: expect (ObjNum 398 snap 96 seq_num 398) 2026-02-20T12:02:58.641 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2011460 2026-02-20T12:02:58.649 INFO:tasks.rados.rados.0.trial153.stdout:887: expect (ObjNum 412 snap 97 seq_num 412) 2026-02-20T12:02:58.718 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2497138 2026-02-20T12:02:58.729 INFO:tasks.rados.rados.0.trial153.stdout:885: finishing write tid 1 to trial15342161-44 2026-02-20T12:02:58.729 INFO:tasks.rados.rados.0.trial153.stdout:886: done (4 left) 2026-02-20T12:02:58.729 INFO:tasks.rados.rados.0.trial153.stdout:887: done (3 left) 2026-02-20T12:02:58.729 INFO:tasks.rados.rados.0.trial153.stdout:890: done (2 left) 2026-02-20T12:02:58.729 INFO:tasks.rados.rados.0.trial153.stdout:891: read oid 21 snap -1 2026-02-20T12:02:58.729 INFO:tasks.rados.rados.0.trial153.stdout:891: expect deleted 2026-02-20T12:02:58.729 INFO:tasks.rados.rados.0.trial153.stdout:892: snap_remove snap 101 2026-02-20T12:02:59.474 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~226s 2026-02-20T12:02:59.567 INFO:tasks.rados.rados.0.trial153.stdout:892: done (3 left) 2026-02-20T12:02:59.567 INFO:tasks.rados.rados.0.trial153.stdout:893: write oid 17 current snap is 107 2026-02-20T12:02:59.568 INFO:tasks.rados.rados.0.trial153.stdout:885: finishing write tid 2 to trial15342161-44 2026-02-20T12:02:59.568 INFO:tasks.rados.rados.0.trial153.stdout:893: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2026-02-20T12:02:59.573 INFO:tasks.rados.rados.0.trial153.stdout:893: writing trial15342161-17 from 561466 to 1352649 tid 1 2026-02-20T12:02:59.578 INFO:tasks.rados.rados.0.trial153.stdout:893: writing trial15342161-17 from 2081113 to 2673917 tid 2 2026-02-20T12:02:59.581 INFO:tasks.rados.rados.0.trial153.stdout:893: writing trial15342161-17 from 3095506 to 3361466 tid 3 2026-02-20T12:02:59.581 INFO:tasks.rados.rados.0.trial153.stdout:894: snap_remove snap 98 2026-02-20T12:03:00.070 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:00.148 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:00.142+0000 7fc464acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:00.150 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:00.146+0000 7fc464acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:00.177 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:00.193 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:00.193 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:00.572 INFO:tasks.rados.rados.0.trial153.stdout:894: done (4 left) 2026-02-20T12:03:00.572 INFO:tasks.rados.rados.0.trial153.stdout:895: snap_create 2026-02-20T12:03:00.572 INFO:tasks.rados.rados.0.trial153.stdout:885: finishing write tid 3 to trial15342161-44 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:885: finishing write tid 4 to trial15342161-44 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:885: finishing write tid 5 to trial15342161-44 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:885: oid 44 updating version 0 to 155 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:885: oid 44 version 155 is already newer than 155 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:885: oid 44 version 155 is already newer than 155 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:885: oid 44 version 155 is already newer than 155 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 44 v 155 (ObjNum 435 snap 107 seq_num 435) dirty exists 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:885: left oid 44 (ObjNum 435 snap 107 seq_num 435) 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:889: finishing write tid 1 to trial15342161-45 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:889: finishing write tid 2 to trial15342161-45 2026-02-20T12:03:00.573 INFO:tasks.rados.rados.0.trial153.stdout:889: finishing write tid 3 to trial15342161-45 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:889: finishing write tid 4 to trial15342161-45 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:889: finishing write tid 5 to trial15342161-45 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:889: oid 45 updating version 0 to 81 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:889: oid 45 version 81 is already newer than 81 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:889: oid 45 version 81 is already newer than 81 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:889: oid 45 version 81 is already newer than 81 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 81 (ObjNum 436 snap 107 seq_num 436) dirty exists 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:889: left oid 45 (ObjNum 436 snap 107 seq_num 436) 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:893: finishing write tid 1 to trial15342161-17 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:893: finishing write tid 2 to trial15342161-17 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:893: finishing write tid 3 to trial15342161-17 2026-02-20T12:03:00.574 INFO:tasks.rados.rados.0.trial153.stdout:893: finishing write tid 4 to trial15342161-17 2026-02-20T12:03:00.575 INFO:tasks.rados.rados.0.trial153.stdout:893: finishing write tid 5 to trial15342161-17 2026-02-20T12:03:00.575 INFO:tasks.rados.rados.0.trial153.stdout:893: oid 17 updating version 0 to 194 2026-02-20T12:03:00.575 INFO:tasks.rados.rados.0.trial153.stdout:893: oid 17 updating version 194 to 198 2026-02-20T12:03:00.575 INFO:tasks.rados.rados.0.trial153.stdout:893: oid 17 version 198 is already newer than 198 2026-02-20T12:03:00.575 INFO:tasks.rados.rados.0.trial153.stdout:893: oid 17 version 198 is already newer than 194 2026-02-20T12:03:00.575 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 17 v 198 (ObjNum 438 snap 107 seq_num 438) dirty exists 2026-02-20T12:03:00.575 INFO:tasks.rados.rados.0.trial153.stdout:893: left oid 17 (ObjNum 438 snap 107 seq_num 438) 2026-02-20T12:03:00.738 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:00.810 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:00.813 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:01.573 INFO:tasks.rados.rados.0.trial153.stdout:885: done (4 left) 2026-02-20T12:03:01.573 INFO:tasks.rados.rados.0.trial153.stdout:889: done (3 left) 2026-02-20T12:03:01.574 INFO:tasks.rados.rados.0.trial153.stdout:891: done (2 left) 2026-02-20T12:03:01.574 INFO:tasks.rados.rados.0.trial153.stdout:893: done (1 left) 2026-02-20T12:03:01.574 INFO:tasks.rados.rados.0.trial153.stdout:895: done (0 left) 2026-02-20T12:03:01.574 INFO:tasks.rados.rados.0.trial153.stdout:896: snap_create 2026-02-20T12:03:02.195 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:02.273 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:02.267+0000 7fe70488b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:02.276 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:02.271+0000 7fe70488b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:02.302 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:02.319 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:02.579 INFO:tasks.rados.rados.0.trial153.stdout:896: done (0 left) 2026-02-20T12:03:02.579 INFO:tasks.rados.rados.0.trial153.stdout:897: write oid 23 current snap is 109 2026-02-20T12:03:02.579 INFO:tasks.rados.rados.0.trial153.stdout:897: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2026-02-20T12:03:02.583 INFO:tasks.rados.rados.0.trial153.stdout:897: writing trial15342161-23 from 578273 to 1044705 tid 1 2026-02-20T12:03:02.588 INFO:tasks.rados.rados.0.trial153.stdout:897: writing trial15342161-23 from 1739595 to 2192410 tid 2 2026-02-20T12:03:02.594 INFO:tasks.rados.rados.0.trial153.stdout:897: writing trial15342161-23 from 2722929 to 3378273 tid 3 2026-02-20T12:03:02.594 INFO:tasks.rados.rados.0.trial153.stdout:898: snap_create 2026-02-20T12:03:02.626 INFO:tasks.rados.rados.0.trial153.stdout:897: finishing write tid 1 to trial15342161-23 2026-02-20T12:03:02.634 INFO:tasks.rados.rados.0.trial153.stdout:897: finishing write tid 2 to trial15342161-23 2026-02-20T12:03:02.642 INFO:tasks.rados.rados.0.trial153.stdout:897: finishing write tid 3 to trial15342161-23 2026-02-20T12:03:02.644 INFO:tasks.rados.rados.0.trial153.stdout:897: finishing write tid 4 to trial15342161-23 2026-02-20T12:03:02.645 INFO:tasks.rados.rados.0.trial153.stdout:897: finishing write tid 5 to trial15342161-23 2026-02-20T12:03:02.646 INFO:tasks.rados.rados.0.trial153.stdout:897: oid 23 updating version 0 to 180 2026-02-20T12:03:02.646 INFO:tasks.rados.rados.0.trial153.stdout:897: oid 23 version 180 is already newer than 180 2026-02-20T12:03:02.646 INFO:tasks.rados.rados.0.trial153.stdout:897: oid 23 version 180 is already newer than 180 2026-02-20T12:03:02.646 INFO:tasks.rados.rados.0.trial153.stdout:897: oid 23 version 180 is already newer than 180 2026-02-20T12:03:02.646 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 23 v 180 (ObjNum 439 snap 109 seq_num 439) dirty exists 2026-02-20T12:03:02.646 INFO:tasks.rados.rados.0.trial153.stdout:897: left oid 23 (ObjNum 439 snap 109 seq_num 439) 2026-02-20T12:03:03.582 INFO:tasks.rados.rados.0.trial153.stdout:897: done (1 left) 2026-02-20T12:03:03.582 INFO:tasks.rados.rados.0.trial153.stdout:898: done (0 left) 2026-02-20T12:03:03.582 INFO:tasks.rados.rados.0.trial153.stdout:899: snap_remove snap 108 2026-02-20T12:03:04.320 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:04.403 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:04.397+0000 7f5c7eecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:04.405 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:04.401+0000 7f5c7eecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:04.432 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:04.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:04.585 INFO:tasks.rados.rados.0.trial153.stdout:899: done (0 left) 2026-02-20T12:03:04.585 INFO:tasks.rados.rados.0.trial153.stdout:900: read oid 15 snap 107 2026-02-20T12:03:04.585 INFO:tasks.rados.rados.0.trial153.stdout:900: expect deleted 2026-02-20T12:03:04.585 INFO:tasks.rados.rados.0.trial153.stdout:901: write (excl) oid 1 current snap is 110 2026-02-20T12:03:04.585 INFO:tasks.rados.rados.0.trial153.stdout:901: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2026-02-20T12:03:04.589 INFO:tasks.rados.rados.0.trial153.stdout:901: writing trial15342161-1 from 595080 to 1136761 tid 1 2026-02-20T12:03:04.594 INFO:tasks.rados.rados.0.trial153.stdout:901: writing trial15342161-1 from 1798077 to 2394550 tid 2 2026-02-20T12:03:04.598 INFO:tasks.rados.rados.0.trial153.stdout:901: writing trial15342161-1 from 3150352 to 3395080 tid 3 2026-02-20T12:03:04.598 INFO:tasks.rados.rados.0.trial153.stdout:902: write oid 19 current snap is 110 2026-02-20T12:03:04.598 INFO:tasks.rados.rados.0.trial153.stdout:902: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2026-02-20T12:03:04.603 INFO:tasks.rados.rados.0.trial153.stdout:902: writing trial15342161-19 from 611887 to 1345170 tid 1 2026-02-20T12:03:04.607 INFO:tasks.rados.rados.0.trial153.stdout:902: writing trial15342161-19 from 1972912 to 2429396 tid 2 2026-02-20T12:03:04.611 INFO:tasks.rados.rados.0.trial153.stdout:902: writing trial15342161-19 from 2894128 to 3411887 tid 3 2026-02-20T12:03:04.612 INFO:tasks.rados.rados.0.trial153.stdout:903: write oid 3 current snap is 110 2026-02-20T12:03:04.612 INFO:tasks.rados.rados.0.trial153.stdout:903: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2026-02-20T12:03:04.615 INFO:tasks.rados.rados.0.trial153.stdout:903: writing trial15342161-3 from 628694 to 1037226 tid 1 2026-02-20T12:03:04.620 INFO:tasks.rados.rados.0.trial153.stdout:903: writing trial15342161-3 from 1515041 to 2115183 tid 2 2026-02-20T12:03:04.624 INFO:tasks.rados.rados.0.trial153.stdout:903: writing trial15342161-3 from 2805198 to 3357310 tid 3 2026-02-20T12:03:04.625 INFO:tasks.rados.rados.0.trial153.stdout:904: snap_create 2026-02-20T12:03:04.628 INFO:tasks.rados.rados.0.trial153.stdout:901: finishing write tid 1 to trial15342161-1 2026-02-20T12:03:04.649 INFO:tasks.rados.rados.0.trial153.stdout:903: finishing write tid 1 to trial15342161-3 2026-02-20T12:03:04.650 INFO:tasks.rados.rados.0.trial153.stdout:902: finishing write tid 1 to trial15342161-19 2026-02-20T12:03:04.650 INFO:tasks.rados.rados.0.trial153.stdout:901: finishing write tid 2 to trial15342161-1 2026-02-20T12:03:04.658 INFO:tasks.rados.rados.0.trial153.stdout:901: finishing write tid 3 to trial15342161-1 2026-02-20T12:03:04.658 INFO:tasks.rados.rados.0.trial153.stdout:902: finishing write tid 2 to trial15342161-19 2026-02-20T12:03:04.659 INFO:tasks.rados.rados.0.trial153.stdout:903: finishing write tid 2 to trial15342161-3 2026-02-20T12:03:04.662 INFO:tasks.rados.rados.0.trial153.stdout:901: finishing write tid 4 to trial15342161-1 2026-02-20T12:03:04.664 INFO:tasks.rados.rados.0.trial153.stdout:901: finishing write tid 5 to trial15342161-1 2026-02-20T12:03:04.664 INFO:tasks.rados.rados.0.trial153.stdout:901: oid 1 updating version 0 to 272 2026-02-20T12:03:04.664 INFO:tasks.rados.rados.0.trial153.stdout:901: oid 1 version 272 is already newer than 272 2026-02-20T12:03:04.664 INFO:tasks.rados.rados.0.trial153.stdout:901: oid 1 version 272 is already newer than 272 2026-02-20T12:03:04.664 INFO:tasks.rados.rados.0.trial153.stdout:901: oid 1 version 272 is already newer than 269 2026-02-20T12:03:04.665 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 1 v 272 (ObjNum 440 snap 110 seq_num 440) dirty exists 2026-02-20T12:03:04.665 INFO:tasks.rados.rados.0.trial153.stdout:901: left oid 1 (ObjNum 440 snap 110 seq_num 440) 2026-02-20T12:03:04.666 INFO:tasks.rados.rados.0.trial153.stdout:903: finishing write tid 3 to trial15342161-3 2026-02-20T12:03:04.666 INFO:tasks.rados.rados.0.trial153.stdout:902: finishing write tid 3 to trial15342161-19 2026-02-20T12:03:04.669 INFO:tasks.rados.rados.0.trial153.stdout:902: finishing write tid 4 to trial15342161-19 2026-02-20T12:03:04.670 INFO:tasks.rados.rados.0.trial153.stdout:903: finishing write tid 4 to trial15342161-3 2026-02-20T12:03:04.671 INFO:tasks.rados.rados.0.trial153.stdout:902: finishing write tid 5 to trial15342161-19 2026-02-20T12:03:04.671 INFO:tasks.rados.rados.0.trial153.stdout:902: oid 19 updating version 0 to 253 2026-02-20T12:03:04.671 INFO:tasks.rados.rados.0.trial153.stdout:902: oid 19 version 253 is already newer than 253 2026-02-20T12:03:04.671 INFO:tasks.rados.rados.0.trial153.stdout:902: oid 19 version 253 is already newer than 253 2026-02-20T12:03:04.672 INFO:tasks.rados.rados.0.trial153.stdout:902: oid 19 version 253 is already newer than 253 2026-02-20T12:03:04.672 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 253 (ObjNum 441 snap 110 seq_num 441) dirty exists 2026-02-20T12:03:04.672 INFO:tasks.rados.rados.0.trial153.stdout:902: left oid 19 (ObjNum 441 snap 110 seq_num 441) 2026-02-20T12:03:04.672 INFO:tasks.rados.rados.0.trial153.stdout:903: finishing write tid 5 to trial15342161-3 2026-02-20T12:03:04.672 INFO:tasks.rados.rados.0.trial153.stdout:903: oid 3 updating version 0 to 129 2026-02-20T12:03:04.672 INFO:tasks.rados.rados.0.trial153.stdout:903: oid 3 version 129 is already newer than 129 2026-02-20T12:03:04.673 INFO:tasks.rados.rados.0.trial153.stdout:903: oid 3 version 129 is already newer than 129 2026-02-20T12:03:04.673 INFO:tasks.rados.rados.0.trial153.stdout:903: oid 3 version 129 is already newer than 129 2026-02-20T12:03:04.673 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 3 v 129 (ObjNum 442 snap 110 seq_num 442) dirty exists 2026-02-20T12:03:04.673 INFO:tasks.rados.rados.0.trial153.stdout:903: left oid 3 (ObjNum 442 snap 110 seq_num 442) 2026-02-20T12:03:04.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~231s 2026-02-20T12:03:05.589 INFO:tasks.rados.rados.0.trial153.stdout:900: done (4 left) 2026-02-20T12:03:05.590 INFO:tasks.rados.rados.0.trial153.stdout:901: done (3 left) 2026-02-20T12:03:05.590 INFO:tasks.rados.rados.0.trial153.stdout:902: done (2 left) 2026-02-20T12:03:05.590 INFO:tasks.rados.rados.0.trial153.stdout:903: done (1 left) 2026-02-20T12:03:05.590 INFO:tasks.rados.rados.0.trial153.stdout:904: done (0 left) 2026-02-20T12:03:05.590 INFO:tasks.rados.rados.0.trial153.stdout:905: snap_remove snap 109 2026-02-20T12:03:05.814 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:05.887 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:05.891 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:06.449 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:06.525 INFO:tasks.rados.rados.0.trial153.stdout:905: done (0 left) 2026-02-20T12:03:06.525 INFO:tasks.rados.rados.0.trial153.stdout:906: write (excl) oid 32 current snap is 111 2026-02-20T12:03:06.525 INFO:tasks.rados.rados.0.trial153.stdout:906: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2026-02-20T12:03:06.529 INFO:tasks.rados.rados.0.trial153.stdout:906: writing trial15342161-32 from 645501 to 1129282 tid 1 2026-02-20T12:03:06.530 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:06.522+0000 7fa08bf9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:06.533 INFO:tasks.rados.rados.0.trial153.stdout:906: writing trial15342161-32 from 1573523 to 2033676 tid 2 2026-02-20T12:03:06.534 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:06.529+0000 7fa08bf9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:06.538 INFO:tasks.rados.rados.0.trial153.stdout:906: writing trial15342161-32 from 2548974 to 3312358 tid 3 2026-02-20T12:03:06.538 INFO:tasks.rados.rados.0.trial153.stdout:907: snap_remove snap 107 2026-02-20T12:03:06.572 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:06.588 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:07.529 INFO:tasks.rados.rados.0.trial153.stdout:907: done (1 left) 2026-02-20T12:03:07.529 INFO:tasks.rados.rados.0.trial153.stdout:908: snap_remove snap 100 2026-02-20T12:03:08.530 INFO:tasks.rados.rados.0.trial153.stdout:908: done (1 left) 2026-02-20T12:03:08.531 INFO:tasks.rados.rados.0.trial153.stdout:909: delete oid 37 current snap is 111 2026-02-20T12:03:08.531 INFO:tasks.rados.rados.0.trial153.stdout:906: finishing write tid 1 to trial15342161-32 2026-02-20T12:03:08.531 INFO:tasks.rados.rados.0.trial153.stdout:906: finishing write tid 2 to trial15342161-32 2026-02-20T12:03:08.531 INFO:tasks.rados.rados.0.trial153.stdout:906: finishing write tid 3 to trial15342161-32 2026-02-20T12:03:08.531 INFO:tasks.rados.rados.0.trial153.stdout:906: finishing write tid 4 to trial15342161-32 2026-02-20T12:03:08.531 INFO:tasks.rados.rados.0.trial153.stdout:906: finishing write tid 5 to trial15342161-32 2026-02-20T12:03:08.531 INFO:tasks.rados.rados.0.trial153.stdout:906: oid 32 updating version 0 to 133 2026-02-20T12:03:08.531 INFO:tasks.rados.rados.0.trial153.stdout:906: oid 32 version 133 is already newer than 133 2026-02-20T12:03:08.531 INFO:tasks.rados.rados.0.trial153.stdout:906: oid 32 version 133 is already newer than 133 2026-02-20T12:03:08.532 INFO:tasks.rados.rados.0.trial153.stdout:906: oid 32 version 133 is already newer than 130 2026-02-20T12:03:08.532 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 32 v 133 (ObjNum 443 snap 111 seq_num 443) dirty exists 2026-02-20T12:03:08.532 INFO:tasks.rados.rados.0.trial153.stdout:906: left oid 32 (ObjNum 443 snap 111 seq_num 443) 2026-02-20T12:03:08.589 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:08.613 INFO:tasks.rados.rados.0.trial153.stdout:906: done (1 left) 2026-02-20T12:03:08.614 INFO:tasks.rados.rados.0.trial153.stdout:909: done (0 left) 2026-02-20T12:03:08.614 INFO:tasks.rados.rados.0.trial153.stdout:910: write oid 25 current snap is 111 2026-02-20T12:03:08.614 INFO:tasks.rados.rados.0.trial153.stdout:910: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2026-02-20T12:03:08.618 INFO:tasks.rados.rados.0.trial153.stdout:910: writing trial15342161-25 from 679115 to 1313394 tid 1 2026-02-20T12:03:08.623 INFO:tasks.rados.rados.0.trial153.stdout:910: writing trial15342161-25 from 2090487 to 2554309 tid 2 2026-02-20T12:03:08.627 INFO:tasks.rados.rados.0.trial153.stdout:910: writing trial15342161-25 from 3003820 to 3479115 tid 3 2026-02-20T12:03:08.627 INFO:tasks.rados.rados.0.trial153.stdout:911: read oid 48 snap -1 2026-02-20T12:03:08.627 INFO:tasks.rados.rados.0.trial153.stdout:911: expect (ObjNum 422 snap 103 seq_num 422) 2026-02-20T12:03:08.627 INFO:tasks.rados.rados.0.trial153.stdout:912: snap_create 2026-02-20T12:03:08.651 INFO:tasks.rados.rados.0.trial153.stdout:911: expect (ObjNum 422 snap 103 seq_num 422) 2026-02-20T12:03:08.658 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1131243 2026-02-20T12:03:08.670 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:08.665+0000 7fcb060c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:08.673 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:08.668+0000 7fcb060c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:08.700 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:08.715 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:08.788 INFO:tasks.rados.rados.0.trial153.stdout:910: finishing write tid 1 to trial15342161-25 2026-02-20T12:03:08.788 INFO:tasks.rados.rados.0.trial153.stdout:910: finishing write tid 2 to trial15342161-25 2026-02-20T12:03:08.788 INFO:tasks.rados.rados.0.trial153.stdout:910: finishing write tid 3 to trial15342161-25 2026-02-20T12:03:08.788 INFO:tasks.rados.rados.0.trial153.stdout:910: finishing write tid 4 to trial15342161-25 2026-02-20T12:03:08.788 INFO:tasks.rados.rados.0.trial153.stdout:910: finishing write tid 5 to trial15342161-25 2026-02-20T12:03:08.788 INFO:tasks.rados.rados.0.trial153.stdout:910: oid 25 updating version 0 to 91 2026-02-20T12:03:08.788 INFO:tasks.rados.rados.0.trial153.stdout:910: oid 25 version 91 is already newer than 91 2026-02-20T12:03:08.789 INFO:tasks.rados.rados.0.trial153.stdout:910: oid 25 version 91 is already newer than 91 2026-02-20T12:03:08.789 INFO:tasks.rados.rados.0.trial153.stdout:910: oid 25 version 91 is already newer than 91 2026-02-20T12:03:08.789 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 25 v 91 (ObjNum 445 snap 111 seq_num 445) dirty exists 2026-02-20T12:03:08.789 INFO:tasks.rados.rados.0.trial153.stdout:910: left oid 25 (ObjNum 445 snap 111 seq_num 445) 2026-02-20T12:03:09.535 INFO:tasks.rados.rados.0.trial153.stdout:910: done (2 left) 2026-02-20T12:03:09.535 INFO:tasks.rados.rados.0.trial153.stdout:911: done (1 left) 2026-02-20T12:03:09.535 INFO:tasks.rados.rados.0.trial153.stdout:912: done (0 left) 2026-02-20T12:03:09.535 INFO:tasks.rados.rados.0.trial153.stdout:913: read oid 31 snap 106 2026-02-20T12:03:09.535 INFO:tasks.rados.rados.0.trial153.stdout:913: expect deleted 2026-02-20T12:03:09.535 INFO:tasks.rados.rados.0.trial153.stdout:914: write oid 25 current snap is 112 2026-02-20T12:03:09.536 INFO:tasks.rados.rados.0.trial153.stdout:914: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2026-02-20T12:03:09.539 INFO:tasks.rados.rados.0.trial153.stdout:914: writing trial15342161-25 from 695922 to 1405450 tid 1 2026-02-20T12:03:09.544 INFO:tasks.rados.rados.0.trial153.stdout:914: writing trial15342161-25 from 2032616 to 2640096 tid 2 2026-02-20T12:03:09.548 INFO:tasks.rados.rados.0.trial153.stdout:914: writing trial15342161-25 from 3198537 to 3495922 tid 3 2026-02-20T12:03:09.548 INFO:tasks.rados.rados.0.trial153.stdout:915: write oid 47 current snap is 112 2026-02-20T12:03:09.548 INFO:tasks.rados.rados.0.trial153.stdout:915: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2026-02-20T12:03:09.553 INFO:tasks.rados.rados.0.trial153.stdout:915: writing trial15342161-47 from 712729 to 1497506 tid 1 2026-02-20T12:03:09.557 INFO:tasks.rados.rados.0.trial153.stdout:915: writing trial15342161-47 from 2091098 to 2558589 tid 2 2026-02-20T12:03:09.561 INFO:tasks.rados.rados.0.trial153.stdout:915: writing trial15342161-47 from 3342313 to 3512729 tid 3 2026-02-20T12:03:09.561 INFO:tasks.rados.rados.0.trial153.stdout:916: snap_remove snap 110 2026-02-20T12:03:10.482 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~237s 2026-02-20T12:03:10.536 INFO:tasks.rados.rados.0.trial153.stdout:916: done (3 left) 2026-02-20T12:03:10.536 INFO:tasks.rados.rados.0.trial153.stdout:917: read oid 50 snap -1 2026-02-20T12:03:10.536 INFO:tasks.rados.rados.0.trial153.stdout:917: expect (ObjNum 400 snap 96 seq_num 400) 2026-02-20T12:03:10.536 INFO:tasks.rados.rados.0.trial153.stdout:918: read oid 20 snap -1 2026-02-20T12:03:10.536 INFO:tasks.rados.rados.0.trial153.stdout:918: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:03:10.537 INFO:tasks.rados.rados.0.trial153.stdout:919: write (excl) oid 2 current snap is 112 2026-02-20T12:03:10.537 INFO:tasks.rados.rados.0.trial153.stdout:919: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2026-02-20T12:03:10.540 INFO:tasks.rados.rados.0.trial153.stdout:919: writing trial15342161-2 from 729536 to 1189562 tid 1 2026-02-20T12:03:10.544 INFO:tasks.rados.rados.0.trial153.stdout:919: writing trial15342161-2 from 1749580 to 2360729 tid 2 2026-02-20T12:03:10.548 INFO:tasks.rados.rados.0.trial153.stdout:919: writing trial15342161-2 from 2853383 to 3305833 tid 3 2026-02-20T12:03:10.548 INFO:tasks.rados.rados.0.trial153.stdout:920: write (excl) oid 10 current snap is 112 2026-02-20T12:03:10.548 INFO:tasks.rados.rados.0.trial153.stdout:920: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2026-02-20T12:03:10.553 INFO:tasks.rados.rados.0.trial153.stdout:920: writing trial15342161-10 from 746343 to 1397971 tid 1 2026-02-20T12:03:10.558 INFO:tasks.rados.rados.0.trial153.stdout:920: writing trial15342161-10 from 1924415 to 2679222 tid 2 2026-02-20T12:03:10.562 INFO:tasks.rados.rados.0.trial153.stdout:920: writing trial15342161-10 from 3397159 to 3546343 tid 3 2026-02-20T12:03:10.562 INFO:tasks.rados.rados.0.trial153.stdout:921: delete oid 46 current snap is 112 2026-02-20T12:03:10.562 INFO:tasks.rados.rados.0.trial153.stdout:915: finishing write tid 1 to trial15342161-47 2026-02-20T12:03:10.562 INFO:tasks.rados.rados.0.trial153.stdout:914: finishing write tid 1 to trial15342161-25 2026-02-20T12:03:10.562 INFO:tasks.rados.rados.0.trial153.stdout:915: finishing write tid 2 to trial15342161-47 2026-02-20T12:03:10.563 INFO:tasks.rados.rados.0.trial153.stdout:914: finishing write tid 2 to trial15342161-25 2026-02-20T12:03:10.563 INFO:tasks.rados.rados.0.trial153.stdout:915: finishing write tid 3 to trial15342161-47 2026-02-20T12:03:10.563 INFO:tasks.rados.rados.0.trial153.stdout:914: finishing write tid 3 to trial15342161-25 2026-02-20T12:03:10.563 INFO:tasks.rados.rados.0.trial153.stdout:915: finishing write tid 4 to trial15342161-47 2026-02-20T12:03:10.563 INFO:tasks.rados.rados.0.trial153.stdout:914: finishing write tid 4 to trial15342161-25 2026-02-20T12:03:10.563 INFO:tasks.rados.rados.0.trial153.stdout:915: finishing write tid 5 to trial15342161-47 2026-02-20T12:03:10.563 INFO:tasks.rados.rados.0.trial153.stdout:915: oid 47 updating version 0 to 185 2026-02-20T12:03:10.563 INFO:tasks.rados.rados.0.trial153.stdout:915: oid 47 version 185 is already newer than 185 2026-02-20T12:03:10.563 INFO:tasks.rados.rados.0.trial153.stdout:915: oid 47 version 185 is already newer than 185 2026-02-20T12:03:10.563 INFO:tasks.rados.rados.0.trial153.stdout:915: oid 47 version 185 is already newer than 185 2026-02-20T12:03:10.564 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 47 v 185 (ObjNum 447 snap 112 seq_num 447) dirty exists 2026-02-20T12:03:10.564 INFO:tasks.rados.rados.0.trial153.stdout:915: left oid 47 (ObjNum 447 snap 112 seq_num 447) 2026-02-20T12:03:10.564 INFO:tasks.rados.rados.0.trial153.stdout:914: finishing write tid 5 to trial15342161-25 2026-02-20T12:03:10.564 INFO:tasks.rados.rados.0.trial153.stdout:914: oid 25 updating version 0 to 96 2026-02-20T12:03:10.564 INFO:tasks.rados.rados.0.trial153.stdout:914: oid 25 version 96 is already newer than 96 2026-02-20T12:03:10.564 INFO:tasks.rados.rados.0.trial153.stdout:914: oid 25 version 96 is already newer than 96 2026-02-20T12:03:10.564 INFO:tasks.rados.rados.0.trial153.stdout:914: oid 25 version 96 is already newer than 96 2026-02-20T12:03:10.564 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 25 v 96 (ObjNum 446 snap 112 seq_num 446) dirty exists 2026-02-20T12:03:10.564 INFO:tasks.rados.rados.0.trial153.stdout:914: left oid 25 (ObjNum 446 snap 112 seq_num 446) 2026-02-20T12:03:10.566 INFO:tasks.rados.rados.0.trial153.stdout:913: done (7 left) 2026-02-20T12:03:10.566 INFO:tasks.rados.rados.0.trial153.stdout:914: done (6 left) 2026-02-20T12:03:10.566 INFO:tasks.rados.rados.0.trial153.stdout:915: done (5 left) 2026-02-20T12:03:10.566 INFO:tasks.rados.rados.0.trial153.stdout:921: done (4 left) 2026-02-20T12:03:10.566 INFO:tasks.rados.rados.0.trial153.stdout:922: write (excl) oid 8 current snap is 112 2026-02-20T12:03:10.566 INFO:tasks.rados.rados.0.trial153.stdout:922: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2026-02-20T12:03:10.571 INFO:tasks.rados.rados.0.trial153.stdout:922: writing trial15342161-8 from 779957 to 1182083 tid 1 2026-02-20T12:03:10.576 INFO:tasks.rados.rados.0.trial153.stdout:922: writing trial15342161-8 from 1925026 to 2683502 tid 2 2026-02-20T12:03:10.580 INFO:tasks.rados.rados.0.trial153.stdout:922: writing trial15342161-8 from 3335652 to 3579957 tid 3 2026-02-20T12:03:10.580 INFO:tasks.rados.rados.0.trial153.stdout:923: delete oid 5 current snap is 112 2026-02-20T12:03:10.580 INFO:tasks.rados.rados.0.trial153.stdout:917: expect (ObjNum 400 snap 96 seq_num 400) 2026-02-20T12:03:10.599 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2242456 2026-02-20T12:03:10.626 INFO:tasks.rados.rados.0.trial153.stdout:920: finishing write tid 1 to trial15342161-10 2026-02-20T12:03:10.626 INFO:tasks.rados.rados.0.trial153.stdout:918: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:03:10.679 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:10.716 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:10.733 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:10.786 INFO:tasks.rados.rados.0.trial153.stdout:919: finishing write tid 1 to trial15342161-2 2026-02-20T12:03:10.787 INFO:tasks.rados.rados.0.trial153.stdout:920: finishing write tid 2 to trial15342161-10 2026-02-20T12:03:10.787 INFO:tasks.rados.rados.0.trial153.stdout:919: finishing write tid 2 to trial15342161-2 2026-02-20T12:03:10.787 INFO:tasks.rados.rados.0.trial153.stdout:917: done (5 left) 2026-02-20T12:03:10.787 INFO:tasks.rados.rados.0.trial153.stdout:918: done (4 left) 2026-02-20T12:03:10.787 INFO:tasks.rados.rados.0.trial153.stdout:923: done (3 left) 2026-02-20T12:03:10.787 INFO:tasks.rados.rados.0.trial153.stdout:924: write oid 30 current snap is 112 2026-02-20T12:03:10.787 INFO:tasks.rados.rados.0.trial153.stdout:920: finishing write tid 3 to trial15342161-10 2026-02-20T12:03:10.787 INFO:tasks.rados.rados.0.trial153.stdout:924: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2026-02-20T12:03:10.790 INFO:tasks.rados.rados.0.trial153.stdout:924: writing trial15342161-30 from 413571 to 966195 tid 1 2026-02-20T12:03:10.792 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:10.787+0000 7f6b6f8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:10.795 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:10.791+0000 7f6b6f8cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:10.796 INFO:tasks.rados.rados.0.trial153.stdout:924: writing trial15342161-30 from 1641990 to 2404135 tid 2 2026-02-20T12:03:10.800 INFO:tasks.rados.rados.0.trial153.stdout:924: writing trial15342161-30 from 2990498 to 3415661 tid 3 2026-02-20T12:03:10.800 INFO:tasks.rados.rados.0.trial153.stdout:925: read oid 15 snap -1 2026-02-20T12:03:10.800 INFO:tasks.rados.rados.0.trial153.stdout:925: expect deleted 2026-02-20T12:03:10.800 INFO:tasks.rados.rados.0.trial153.stdout:926: write oid 7 current snap is 112 2026-02-20T12:03:10.800 INFO:tasks.rados.rados.0.trial153.stdout:920: finishing write tid 4 to trial15342161-10 2026-02-20T12:03:10.800 INFO:tasks.rados.rados.0.trial153.stdout:926: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2026-02-20T12:03:10.803 INFO:tasks.rados.rados.0.trial153.stdout:926: writing trial15342161-7 from 430378 to 1058251 tid 1 2026-02-20T12:03:10.808 INFO:tasks.rados.rados.0.trial153.stdout:926: writing trial15342161-7 from 1700472 to 2322628 tid 2 2026-02-20T12:03:10.814 INFO:tasks.rados.rados.0.trial153.stdout:926: writing trial15342161-7 from 3017921 to 3630378 tid 3 2026-02-20T12:03:10.814 INFO:tasks.rados.rados.0.trial153.stdout:927: delete oid 4 current snap is 112 2026-02-20T12:03:10.814 INFO:tasks.rados.rados.0.trial153.stdout:920: finishing write tid 5 to trial15342161-10 2026-02-20T12:03:10.814 INFO:tasks.rados.rados.0.trial153.stdout:920: oid 10 updating version 0 to 159 2026-02-20T12:03:10.814 INFO:tasks.rados.rados.0.trial153.stdout:920: oid 10 version 159 is already newer than 159 2026-02-20T12:03:10.814 INFO:tasks.rados.rados.0.trial153.stdout:920: oid 10 version 159 is already newer than 156 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:920: oid 10 version 159 is already newer than 159 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 10 v 159 (ObjNum 449 snap 112 seq_num 449) dirty exists 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:920: left oid 10 (ObjNum 449 snap 112 seq_num 449) 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:919: finishing write tid 3 to trial15342161-2 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:919: finishing write tid 4 to trial15342161-2 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:919: finishing write tid 5 to trial15342161-2 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:919: oid 2 updating version 0 to 277 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:919: oid 2 version 277 is already newer than 277 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:919: oid 2 version 277 is already newer than 277 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:919: oid 2 version 277 is already newer than 274 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 2 v 277 (ObjNum 448 snap 112 seq_num 448) dirty exists 2026-02-20T12:03:10.815 INFO:tasks.rados.rados.0.trial153.stdout:919: left oid 2 (ObjNum 448 snap 112 seq_num 448) 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:922: finishing write tid 1 to trial15342161-8 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:922: finishing write tid 2 to trial15342161-8 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:922: finishing write tid 3 to trial15342161-8 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:922: finishing write tid 4 to trial15342161-8 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:922: finishing write tid 5 to trial15342161-8 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:922: oid 8 updating version 0 to 92 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:922: oid 8 version 92 is already newer than 92 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:922: oid 8 version 92 is already newer than 92 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:922: oid 8 version 92 is already newer than 89 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 8 v 92 (ObjNum 451 snap 112 seq_num 451) dirty exists 2026-02-20T12:03:10.816 INFO:tasks.rados.rados.0.trial153.stdout:922: left oid 8 (ObjNum 451 snap 112 seq_num 451) 2026-02-20T12:03:10.817 INFO:tasks.rados.rados.0.trial153.stdout:919: done (6 left) 2026-02-20T12:03:10.817 INFO:tasks.rados.rados.0.trial153.stdout:920: done (5 left) 2026-02-20T12:03:10.818 INFO:tasks.rados.rados.0.trial153.stdout:922: done (4 left) 2026-02-20T12:03:10.818 INFO:tasks.rados.rados.0.trial153.stdout:925: done (3 left) 2026-02-20T12:03:10.818 INFO:tasks.rados.rados.0.trial153.stdout:927: done (2 left) 2026-02-20T12:03:10.818 INFO:tasks.rados.rados.0.trial153.stdout:928: read oid 47 snap 111 2026-02-20T12:03:10.818 INFO:tasks.rados.rados.0.trial153.stdout:928: expect (ObjNum 406 snap 97 seq_num 406) 2026-02-20T12:03:10.818 INFO:tasks.rados.rados.0.trial153.stdout:929: snap_remove snap 106 2026-02-20T12:03:10.860 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:10.874 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:10.875 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:10.892 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:10.963 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:10.967 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:11.542 INFO:tasks.rados.rados.0.trial153.stdout:929: done (3 left) 2026-02-20T12:03:11.542 INFO:tasks.rados.rados.0.trial153.stdout:930: read oid 6 snap 111 2026-02-20T12:03:11.542 INFO:tasks.rados.rados.0.trial153.stdout:930: expect deleted 2026-02-20T12:03:11.542 INFO:tasks.rados.rados.0.trial153.stdout:931: read oid 20 snap -1 2026-02-20T12:03:11.542 INFO:tasks.rados.rados.0.trial153.stdout:931: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:03:11.542 INFO:tasks.rados.rados.0.trial153.stdout:932: write (excl) oid 5 current snap is 112 2026-02-20T12:03:11.543 INFO:tasks.rados.rados.0.trial153.stdout:932: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2026-02-20T12:03:11.546 INFO:tasks.rados.rados.0.trial153.stdout:932: writing trial15342161-5 from 463992 to 1242363 tid 1 2026-02-20T12:03:11.550 INFO:tasks.rados.rados.0.trial153.stdout:932: writing trial15342161-5 from 1701083 to 2326908 tid 2 2026-02-20T12:03:11.555 INFO:tasks.rados.rados.0.trial153.stdout:932: writing trial15342161-5 from 2956414 to 3615393 tid 3 2026-02-20T12:03:11.555 INFO:tasks.rados.rados.0.trial153.stdout:933: read oid 28 snap -1 2026-02-20T12:03:11.556 INFO:tasks.rados.rados.0.trial153.stdout:933: expect (ObjNum 402 snap 97 seq_num 402) 2026-02-20T12:03:11.556 INFO:tasks.rados.rados.0.trial153.stdout:934: delete oid 23 current snap is 112 2026-02-20T12:03:11.556 INFO:tasks.rados.rados.0.trial153.stdout:928: expect (ObjNum 406 snap 97 seq_num 406) 2026-02-20T12:03:11.634 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2479413 2026-02-20T12:03:11.640 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:11.689 INFO:tasks.rados.rados.0.trial153.stdout:924: finishing write tid 1 to trial15342161-30 2026-02-20T12:03:11.690 INFO:tasks.rados.rados.0.trial153.stdout:926: finishing write tid 1 to trial15342161-7 2026-02-20T12:03:11.690 INFO:tasks.rados.rados.0.trial153.stdout:928: done (7 left) 2026-02-20T12:03:11.690 INFO:tasks.rados.rados.0.trial153.stdout:934: done (6 left) 2026-02-20T12:03:11.690 INFO:tasks.rados.rados.0.trial153.stdout:935: read oid 24 snap 111 2026-02-20T12:03:11.690 INFO:tasks.rados.rados.0.trial153.stdout:935: expect (ObjNum 424 snap 103 seq_num 424) 2026-02-20T12:03:11.690 INFO:tasks.rados.rados.0.trial153.stdout:936: write oid 50 current snap is 112 2026-02-20T12:03:11.690 INFO:tasks.rados.rados.0.trial153.stdout:926: finishing write tid 2 to trial15342161-7 2026-02-20T12:03:11.690 INFO:tasks.rados.rados.0.trial153.stdout:924: finishing write tid 2 to trial15342161-30 2026-02-20T12:03:11.690 INFO:tasks.rados.rados.0.trial153.stdout:936: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2026-02-20T12:03:11.693 INFO:tasks.rados.rados.0.trial153.stdout:936: writing trial15342161-50 from 497606 to 1142828 tid 1 2026-02-20T12:03:11.699 INFO:tasks.rados.rados.0.trial153.stdout:936: writing trial15342161-50 from 1934400 to 2563894 tid 2 2026-02-20T12:03:11.704 INFO:tasks.rados.rados.0.trial153.stdout:936: writing trial15342161-50 from 3243966 to 3697606 tid 3 2026-02-20T12:03:11.704 INFO:tasks.rados.rados.0.trial153.stdout:937: snap_create 2026-02-20T12:03:11.704 INFO:tasks.rados.rados.0.trial153.stdout:924: finishing write tid 3 to trial15342161-30 2026-02-20T12:03:11.704 INFO:tasks.rados.rados.0.trial153.stdout:926: finishing write tid 3 to trial15342161-7 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:926: finishing write tid 4 to trial15342161-7 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:926: finishing write tid 5 to trial15342161-7 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:926: oid 7 updating version 0 to 85 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:926: oid 7 version 85 is already newer than 85 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:926: oid 7 version 85 is already newer than 85 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:926: oid 7 version 85 is already newer than 85 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 7 v 85 (ObjNum 454 snap 112 seq_num 454) dirty exists 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:926: left oid 7 (ObjNum 454 snap 112 seq_num 454) 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:924: finishing write tid 4 to trial15342161-30 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:924: finishing write tid 5 to trial15342161-30 2026-02-20T12:03:11.705 INFO:tasks.rados.rados.0.trial153.stdout:924: oid 30 updating version 0 to 164 2026-02-20T12:03:11.706 INFO:tasks.rados.rados.0.trial153.stdout:924: oid 30 version 164 is already newer than 164 2026-02-20T12:03:11.706 INFO:tasks.rados.rados.0.trial153.stdout:924: oid 30 version 164 is already newer than 164 2026-02-20T12:03:11.706 INFO:tasks.rados.rados.0.trial153.stdout:924: oid 30 version 164 is already newer than 164 2026-02-20T12:03:11.706 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 30 v 164 (ObjNum 453 snap 112 seq_num 453) dirty exists 2026-02-20T12:03:11.706 INFO:tasks.rados.rados.0.trial153.stdout:924: left oid 30 (ObjNum 453 snap 112 seq_num 453) 2026-02-20T12:03:11.706 INFO:tasks.rados.rados.0.trial153.stdout:932: finishing write tid 1 to trial15342161-5 2026-02-20T12:03:11.706 INFO:tasks.rados.rados.0.trial153.stdout:931: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:03:11.825 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1378033 2026-02-20T12:03:11.856 INFO:tasks.rados.rados.0.trial153.stdout:933: expect (ObjNum 402 snap 97 seq_num 402) 2026-02-20T12:03:11.883 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:11.913 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:11.943 INFO:tasks.rados.rados.0.trial153.stdout:932: finishing write tid 2 to trial15342161-5 2026-02-20T12:03:11.944 INFO:tasks.rados.rados.0.trial153.stdout:932: finishing write tid 3 to trial15342161-5 2026-02-20T12:03:11.944 INFO:tasks.rados.rados.0.trial153.stdout:932: finishing write tid 4 to trial15342161-5 2026-02-20T12:03:11.944 INFO:tasks.rados.rados.0.trial153.stdout:932: finishing write tid 5 to trial15342161-5 2026-02-20T12:03:11.944 INFO:tasks.rados.rados.0.trial153.stdout:932: oid 5 updating version 0 to 98 2026-02-20T12:03:11.944 INFO:tasks.rados.rados.0.trial153.stdout:932: oid 5 version 98 is already newer than 98 2026-02-20T12:03:11.944 INFO:tasks.rados.rados.0.trial153.stdout:932: oid 5 version 98 is already newer than 98 2026-02-20T12:03:11.944 INFO:tasks.rados.rados.0.trial153.stdout:932: oid 5 version 98 is already newer than 95 2026-02-20T12:03:11.944 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 5 v 98 (ObjNum 456 snap 112 seq_num 456) dirty exists 2026-02-20T12:03:11.944 INFO:tasks.rados.rados.0.trial153.stdout:932: left oid 5 (ObjNum 456 snap 112 seq_num 456) 2026-02-20T12:03:11.945 INFO:tasks.rados.rados.0.trial153.stdout:935: expect (ObjNum 424 snap 103 seq_num 424) 2026-02-20T12:03:11.950 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2812731 2026-02-20T12:03:11.956 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 258508 2026-02-20T12:03:12.008 INFO:tasks.rados.rados.0.trial153.stdout:936: finishing write tid 1 to trial15342161-50 2026-02-20T12:03:12.009 INFO:tasks.rados.rados.0.trial153.stdout:936: finishing write tid 2 to trial15342161-50 2026-02-20T12:03:12.009 INFO:tasks.rados.rados.0.trial153.stdout:936: finishing write tid 3 to trial15342161-50 2026-02-20T12:03:12.009 INFO:tasks.rados.rados.0.trial153.stdout:936: finishing write tid 4 to trial15342161-50 2026-02-20T12:03:12.009 INFO:tasks.rados.rados.0.trial153.stdout:936: finishing write tid 5 to trial15342161-50 2026-02-20T12:03:12.009 INFO:tasks.rados.rados.0.trial153.stdout:936: oid 50 updating version 0 to 266 2026-02-20T12:03:12.009 INFO:tasks.rados.rados.0.trial153.stdout:936: oid 50 version 266 is already newer than 266 2026-02-20T12:03:12.009 INFO:tasks.rados.rados.0.trial153.stdout:936: oid 50 version 266 is already newer than 266 2026-02-20T12:03:12.009 INFO:tasks.rados.rados.0.trial153.stdout:936: oid 50 version 266 is already newer than 266 2026-02-20T12:03:12.010 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 50 v 266 (ObjNum 458 snap 112 seq_num 458) dirty exists 2026-02-20T12:03:12.010 INFO:tasks.rados.rados.0.trial153.stdout:936: left oid 50 (ObjNum 458 snap 112 seq_num 458) 2026-02-20T12:03:12.547 INFO:tasks.rados.rados.0.trial153.stdout:924: done (8 left) 2026-02-20T12:03:12.547 INFO:tasks.rados.rados.0.trial153.stdout:926: done (7 left) 2026-02-20T12:03:12.548 INFO:tasks.rados.rados.0.trial153.stdout:930: done (6 left) 2026-02-20T12:03:12.548 INFO:tasks.rados.rados.0.trial153.stdout:931: done (5 left) 2026-02-20T12:03:12.548 INFO:tasks.rados.rados.0.trial153.stdout:932: done (4 left) 2026-02-20T12:03:12.548 INFO:tasks.rados.rados.0.trial153.stdout:933: done (3 left) 2026-02-20T12:03:12.548 INFO:tasks.rados.rados.0.trial153.stdout:935: done (2 left) 2026-02-20T12:03:12.548 INFO:tasks.rados.rados.0.trial153.stdout:936: done (1 left) 2026-02-20T12:03:12.548 INFO:tasks.rados.rados.0.trial153.stdout:937: done (0 left) 2026-02-20T12:03:12.548 INFO:tasks.rados.rados.0.trial153.stdout:938: write oid 32 current snap is 113 2026-02-20T12:03:12.548 INFO:tasks.rados.rados.0.trial153.stdout:938: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2026-02-20T12:03:12.552 INFO:tasks.rados.rados.0.trial153.stdout:938: writing trial15342161-32 from 514413 to 1234884 tid 1 2026-02-20T12:03:12.557 INFO:tasks.rados.rados.0.trial153.stdout:938: writing trial15342161-32 from 1876529 to 2649681 tid 2 2026-02-20T12:03:12.561 INFO:tasks.rados.rados.0.trial153.stdout:938: writing trial15342161-32 from 3438683 to 3714413 tid 3 2026-02-20T12:03:12.562 INFO:tasks.rados.rados.0.trial153.stdout:939: delete oid 34 current snap is 113 2026-02-20T12:03:12.565 INFO:tasks.rados.rados.0.trial153.stdout:939: done (1 left) 2026-02-20T12:03:12.565 INFO:tasks.rados.rados.0.trial153.stdout:940: read oid 20 snap -1 2026-02-20T12:03:12.565 INFO:tasks.rados.rados.0.trial153.stdout:940: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:03:12.565 INFO:tasks.rados.rados.0.trial153.stdout:941: snap_create 2026-02-20T12:03:12.588 INFO:tasks.rados.rados.0.trial153.stdout:940: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:03:12.588 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:12.653 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1346915 2026-02-20T12:03:12.711 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2159916 2026-02-20T12:03:12.733 INFO:tasks.rados.rados.0.trial153.stdout:938: finishing write tid 1 to trial15342161-32 2026-02-20T12:03:12.734 INFO:tasks.rados.rados.0.trial153.stdout:938: finishing write tid 2 to trial15342161-32 2026-02-20T12:03:12.734 INFO:tasks.rados.rados.0.trial153.stdout:938: finishing write tid 3 to trial15342161-32 2026-02-20T12:03:12.734 INFO:tasks.rados.rados.0.trial153.stdout:938: finishing write tid 4 to trial15342161-32 2026-02-20T12:03:12.734 INFO:tasks.rados.rados.0.trial153.stdout:938: finishing write tid 5 to trial15342161-32 2026-02-20T12:03:12.734 INFO:tasks.rados.rados.0.trial153.stdout:938: oid 32 updating version 0 to 138 2026-02-20T12:03:12.734 INFO:tasks.rados.rados.0.trial153.stdout:938: oid 32 version 138 is already newer than 138 2026-02-20T12:03:12.734 INFO:tasks.rados.rados.0.trial153.stdout:938: oid 32 version 138 is already newer than 138 2026-02-20T12:03:12.734 INFO:tasks.rados.rados.0.trial153.stdout:938: oid 32 version 138 is already newer than 138 2026-02-20T12:03:12.734 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 32 v 138 (ObjNum 459 snap 113 seq_num 459) dirty exists 2026-02-20T12:03:12.734 INFO:tasks.rados.rados.0.trial153.stdout:938: left oid 32 (ObjNum 459 snap 113 seq_num 459) 2026-02-20T12:03:12.876 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:12.955 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:12.949+0000 7f1868acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:12.957 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:12.953+0000 7f1868acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:12.984 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:13.000 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:13.551 INFO:tasks.rados.rados.0.trial153.stdout:938: done (2 left) 2026-02-20T12:03:13.551 INFO:tasks.rados.rados.0.trial153.stdout:940: done (1 left) 2026-02-20T12:03:13.551 INFO:tasks.rados.rados.0.trial153.stdout:941: done (0 left) 2026-02-20T12:03:13.552 INFO:tasks.rados.rados.0.trial153.stdout:942: delete oid 7 current snap is 114 2026-02-20T12:03:13.588 INFO:tasks.rados.rados.0.trial153.stdout:942: done (0 left) 2026-02-20T12:03:13.588 INFO:tasks.rados.rados.0.trial153.stdout:943: read oid 22 snap -1 2026-02-20T12:03:13.588 INFO:tasks.rados.rados.0.trial153.stdout:943: expect (ObjNum 408 snap 97 seq_num 408) 2026-02-20T12:03:13.588 INFO:tasks.rados.rados.0.trial153.stdout:944: read oid 46 snap 112 2026-02-20T12:03:13.588 INFO:tasks.rados.rados.0.trial153.stdout:944: expect deleted 2026-02-20T12:03:13.589 INFO:tasks.rados.rados.0.trial153.stdout:945: delete oid 23 current snap is 114 2026-02-20T12:03:13.612 INFO:tasks.rados.rados.0.trial153.stdout:944: done (2 left) 2026-02-20T12:03:13.612 INFO:tasks.rados.rados.0.trial153.stdout:945: done (1 left) 2026-02-20T12:03:13.612 INFO:tasks.rados.rados.0.trial153.stdout:946: read oid 25 snap -1 2026-02-20T12:03:13.612 INFO:tasks.rados.rados.0.trial153.stdout:946: expect (ObjNum 446 snap 112 seq_num 446) 2026-02-20T12:03:13.612 INFO:tasks.rados.rados.0.trial153.stdout:947: write oid 18 current snap is 114 2026-02-20T12:03:13.612 INFO:tasks.rados.rados.0.trial153.stdout:947: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2026-02-20T12:03:13.617 INFO:tasks.rados.rados.0.trial153.stdout:947: writing trial15342161-18 from 581641 to 1203108 tid 1 2026-02-20T12:03:13.623 INFO:tasks.rados.rados.0.trial153.stdout:947: writing trial15342161-18 from 1994104 to 2774594 tid 2 2026-02-20T12:03:13.626 INFO:tasks.rados.rados.0.trial153.stdout:947: writing trial15342161-18 from 3432022 to 3781641 tid 3 2026-02-20T12:03:13.627 INFO:tasks.rados.rados.0.trial153.stdout:948: snap_remove snap 111 2026-02-20T12:03:14.554 INFO:tasks.rados.rados.0.trial153.stdout:948: done (3 left) 2026-02-20T12:03:14.554 INFO:tasks.rados.rados.0.trial153.stdout:949: write oid 30 current snap is 114 2026-02-20T12:03:14.554 INFO:tasks.rados.rados.0.trial153.stdout:949: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2026-02-20T12:03:14.558 INFO:tasks.rados.rados.0.trial153.stdout:949: writing trial15342161-30 from 598448 to 1011517 tid 1 2026-02-20T12:03:14.562 INFO:tasks.rados.rados.0.trial153.stdout:949: writing trial15342161-30 from 1768939 to 2293087 tid 2 2026-02-20T12:03:14.566 INFO:tasks.rados.rados.0.trial153.stdout:949: writing trial15342161-30 from 2775798 to 3357659 tid 3 2026-02-20T12:03:14.569 INFO:tasks.rados.rados.0.trial153.stdout:949: writing trial15342161-30 from 3781488 to 3798448 tid 4 2026-02-20T12:03:14.569 INFO:tasks.rados.rados.0.trial153.stdout:950: read oid 8 snap -1 2026-02-20T12:03:14.569 INFO:tasks.rados.rados.0.trial153.stdout:950: expect (ObjNum 451 snap 112 seq_num 451) 2026-02-20T12:03:14.569 INFO:tasks.rados.rados.0.trial153.stdout:951: snap_create 2026-02-20T12:03:14.569 INFO:tasks.rados.rados.0.trial153.stdout:943: expect (ObjNum 408 snap 97 seq_num 408) 2026-02-20T12:03:14.602 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:14.637 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 10611 2026-02-20T12:03:14.670 INFO:tasks.rados.rados.0.trial153.stdout:946: expect (ObjNum 446 snap 112 seq_num 446) 2026-02-20T12:03:14.703 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2734876 2026-02-20T12:03:14.724 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:14.850 INFO:tasks.rados.rados.0.trial153.stdout:947: finishing write tid 1 to trial15342161-18 2026-02-20T12:03:14.850 INFO:tasks.rados.rados.0.trial153.stdout:947: finishing write tid 2 to trial15342161-18 2026-02-20T12:03:14.850 INFO:tasks.rados.rados.0.trial153.stdout:947: finishing write tid 3 to trial15342161-18 2026-02-20T12:03:14.850 INFO:tasks.rados.rados.0.trial153.stdout:947: finishing write tid 4 to trial15342161-18 2026-02-20T12:03:14.850 INFO:tasks.rados.rados.0.trial153.stdout:947: finishing write tid 5 to trial15342161-18 2026-02-20T12:03:14.850 INFO:tasks.rados.rados.0.trial153.stdout:947: oid 18 updating version 0 to 74 2026-02-20T12:03:14.851 INFO:tasks.rados.rados.0.trial153.stdout:947: oid 18 version 74 is already newer than 74 2026-02-20T12:03:14.851 INFO:tasks.rados.rados.0.trial153.stdout:947: oid 18 version 74 is already newer than 74 2026-02-20T12:03:14.851 INFO:tasks.rados.rados.0.trial153.stdout:947: oid 18 version 74 is already newer than 74 2026-02-20T12:03:14.851 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 18 v 74 (ObjNum 463 snap 114 seq_num 463) dirty exists 2026-02-20T12:03:14.851 INFO:tasks.rados.rados.0.trial153.stdout:947: left oid 18 (ObjNum 463 snap 114 seq_num 463) 2026-02-20T12:03:14.851 INFO:tasks.rados.rados.0.trial153.stdout:950: expect (ObjNum 451 snap 112 seq_num 451) 2026-02-20T12:03:15.000 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:15.001 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:15.065 INFO:tasks.rados.rados.0.trial153.stdout:949: finishing write tid 1 to trial15342161-30 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:949: finishing write tid 2 to trial15342161-30 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:949: finishing write tid 3 to trial15342161-30 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:949: finishing write tid 4 to trial15342161-30 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:949: finishing write tid 5 to trial15342161-30 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:949: finishing write tid 6 to trial15342161-30 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:949: oid 30 updating version 0 to 170 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:949: oid 30 version 170 is already newer than 170 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:949: oid 30 version 170 is already newer than 170 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:949: oid 30 version 170 is already newer than 170 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:949: oid 30 version 170 is already newer than 170 2026-02-20T12:03:15.066 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 30 v 170 (ObjNum 464 snap 114 seq_num 464) dirty exists 2026-02-20T12:03:15.067 INFO:tasks.rados.rados.0.trial153.stdout:949: left oid 30 (ObjNum 464 snap 114 seq_num 464) 2026-02-20T12:03:15.079 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:15.073+0000 7f7a7b3b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:15.081 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:15.077+0000 7f7a7b3b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:15.108 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:15.123 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:15.559 INFO:tasks.rados.rados.0.trial153.stdout:943: done (5 left) 2026-02-20T12:03:15.559 INFO:tasks.rados.rados.0.trial153.stdout:946: done (4 left) 2026-02-20T12:03:15.559 INFO:tasks.rados.rados.0.trial153.stdout:947: done (3 left) 2026-02-20T12:03:15.559 INFO:tasks.rados.rados.0.trial153.stdout:949: done (2 left) 2026-02-20T12:03:15.559 INFO:tasks.rados.rados.0.trial153.stdout:950: done (1 left) 2026-02-20T12:03:15.560 INFO:tasks.rados.rados.0.trial153.stdout:951: done (0 left) 2026-02-20T12:03:15.560 INFO:tasks.rados.rados.0.trial153.stdout:952: snap_remove snap 114 2026-02-20T12:03:15.968 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:15.987 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~242s 2026-02-20T12:03:16.041 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:16.044 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:16.528 INFO:tasks.rados.rados.0.trial153.stdout:952: done (0 left) 2026-02-20T12:03:16.529 INFO:tasks.rados.rados.0.trial153.stdout:953: delete oid 3 current snap is 115 2026-02-20T12:03:16.571 INFO:tasks.rados.rados.0.trial153.stdout:953: done (0 left) 2026-02-20T12:03:16.571 INFO:tasks.rados.rados.0.trial153.stdout:954: write oid 14 current snap is 115 2026-02-20T12:03:16.571 INFO:tasks.rados.rados.0.trial153.stdout:954: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2026-02-20T12:03:16.576 INFO:tasks.rados.rados.0.trial153.stdout:954: writing trial15342161-14 from 632062 to 1195629 tid 1 2026-02-20T12:03:16.580 INFO:tasks.rados.rados.0.trial153.stdout:954: writing trial15342161-14 from 1885903 to 2413720 tid 2 2026-02-20T12:03:16.585 INFO:tasks.rados.rados.0.trial153.stdout:954: writing trial15342161-14 from 2830644 to 3551402 tid 3 2026-02-20T12:03:16.585 INFO:tasks.rados.rados.0.trial153.stdout:955: read oid 27 snap 113 2026-02-20T12:03:16.585 INFO:tasks.rados.rados.0.trial153.stdout:955: expect deleted 2026-02-20T12:03:16.585 INFO:tasks.rados.rados.0.trial153.stdout:956: snap_remove snap 112 2026-02-20T12:03:17.124 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:17.201 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:17.196+0000 7efe34ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:17.204 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:17.200+0000 7efe34ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:17.231 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:17.247 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:17.564 INFO:tasks.rados.rados.0.trial153.stdout:956: done (2 left) 2026-02-20T12:03:17.564 INFO:tasks.rados.rados.0.trial153.stdout:957: write (excl) oid 19 current snap is 115 2026-02-20T12:03:17.564 INFO:tasks.rados.rados.0.trial153.stdout:957: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2026-02-20T12:03:17.568 INFO:tasks.rados.rados.0.trial153.stdout:957: writing trial15342161-19 from 648869 to 1287685 tid 1 2026-02-20T12:03:17.573 INFO:tasks.rados.rados.0.trial153.stdout:957: writing trial15342161-19 from 1828032 to 2615860 tid 2 2026-02-20T12:03:17.578 INFO:tasks.rados.rados.0.trial153.stdout:957: writing trial15342161-19 from 3141714 to 3673744 tid 3 2026-02-20T12:03:17.578 INFO:tasks.rados.rados.0.trial153.stdout:958: snap_create 2026-02-20T12:03:17.578 INFO:tasks.rados.rados.0.trial153.stdout:954: finishing write tid 1 to trial15342161-14 2026-02-20T12:03:17.578 INFO:tasks.rados.rados.0.trial153.stdout:954: finishing write tid 2 to trial15342161-14 2026-02-20T12:03:17.578 INFO:tasks.rados.rados.0.trial153.stdout:954: finishing write tid 3 to trial15342161-14 2026-02-20T12:03:17.578 INFO:tasks.rados.rados.0.trial153.stdout:954: finishing write tid 4 to trial15342161-14 2026-02-20T12:03:17.578 INFO:tasks.rados.rados.0.trial153.stdout:954: finishing write tid 5 to trial15342161-14 2026-02-20T12:03:17.578 INFO:tasks.rados.rados.0.trial153.stdout:954: oid 14 updating version 0 to 273 2026-02-20T12:03:17.579 INFO:tasks.rados.rados.0.trial153.stdout:954: oid 14 version 273 is already newer than 273 2026-02-20T12:03:17.579 INFO:tasks.rados.rados.0.trial153.stdout:954: oid 14 version 273 is already newer than 273 2026-02-20T12:03:17.579 INFO:tasks.rados.rados.0.trial153.stdout:954: oid 14 version 273 is already newer than 273 2026-02-20T12:03:17.579 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 273 (ObjNum 466 snap 115 seq_num 466) dirty exists 2026-02-20T12:03:17.579 INFO:tasks.rados.rados.0.trial153.stdout:954: left oid 14 (ObjNum 466 snap 115 seq_num 466) 2026-02-20T12:03:17.617 INFO:tasks.rados.rados.0.trial153.stdout:957: finishing write tid 1 to trial15342161-19 2026-02-20T12:03:17.633 INFO:tasks.rados.rados.0.trial153.stdout:957: finishing write tid 2 to trial15342161-19 2026-02-20T12:03:17.640 INFO:tasks.rados.rados.0.trial153.stdout:957: finishing write tid 3 to trial15342161-19 2026-02-20T12:03:17.644 INFO:tasks.rados.rados.0.trial153.stdout:957: finishing write tid 4 to trial15342161-19 2026-02-20T12:03:17.646 INFO:tasks.rados.rados.0.trial153.stdout:957: finishing write tid 5 to trial15342161-19 2026-02-20T12:03:17.646 INFO:tasks.rados.rados.0.trial153.stdout:957: oid 19 updating version 0 to 277 2026-02-20T12:03:17.646 INFO:tasks.rados.rados.0.trial153.stdout:957: oid 19 updating version 277 to 280 2026-02-20T12:03:17.646 INFO:tasks.rados.rados.0.trial153.stdout:957: oid 19 version 280 is already newer than 280 2026-02-20T12:03:17.646 INFO:tasks.rados.rados.0.trial153.stdout:957: oid 19 version 280 is already newer than 280 2026-02-20T12:03:17.647 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 280 (ObjNum 467 snap 115 seq_num 467) dirty exists 2026-02-20T12:03:17.647 INFO:tasks.rados.rados.0.trial153.stdout:957: left oid 19 (ObjNum 467 snap 115 seq_num 467) 2026-02-20T12:03:18.567 INFO:tasks.rados.rados.0.trial153.stdout:954: done (3 left) 2026-02-20T12:03:18.567 INFO:tasks.rados.rados.0.trial153.stdout:955: done (2 left) 2026-02-20T12:03:18.567 INFO:tasks.rados.rados.0.trial153.stdout:957: done (1 left) 2026-02-20T12:03:18.567 INFO:tasks.rados.rados.0.trial153.stdout:958: done (0 left) 2026-02-20T12:03:18.568 INFO:tasks.rados.rados.0.trial153.stdout:959: write oid 32 current snap is 116 2026-02-20T12:03:18.568 INFO:tasks.rados.rados.0.trial153.stdout:959: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2026-02-20T12:03:18.572 INFO:tasks.rados.rados.0.trial153.stdout:959: writing trial15342161-32 from 665676 to 1379741 tid 1 2026-02-20T12:03:18.576 INFO:tasks.rados.rados.0.trial153.stdout:959: writing trial15342161-32 from 1886514 to 2418000 tid 2 2026-02-20T12:03:18.582 INFO:tasks.rados.rados.0.trial153.stdout:959: writing trial15342161-32 from 3169137 to 3865676 tid 3 2026-02-20T12:03:18.582 INFO:tasks.rados.rados.0.trial153.stdout:960: write oid 28 current snap is 116 2026-02-20T12:03:18.582 INFO:tasks.rados.rados.0.trial153.stdout:960: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2026-02-20T12:03:18.587 INFO:tasks.rados.rados.0.trial153.stdout:960: writing trial15342161-28 from 682483 to 1471797 tid 1 2026-02-20T12:03:18.592 INFO:tasks.rados.rados.0.trial153.stdout:960: writing trial15342161-28 from 1944996 to 2736493 tid 2 2026-02-20T12:03:18.596 INFO:tasks.rados.rados.0.trial153.stdout:960: writing trial15342161-28 from 3196560 to 3751134 tid 3 2026-02-20T12:03:18.596 INFO:tasks.rados.rados.0.trial153.stdout:961: read oid 26 snap 113 2026-02-20T12:03:18.596 INFO:tasks.rados.rados.0.trial153.stdout:961: expect (ObjNum 332 snap 80 seq_num 332) 2026-02-20T12:03:18.597 INFO:tasks.rados.rados.0.trial153.stdout:962: delete oid 7 current snap is 116 2026-02-20T12:03:18.600 INFO:tasks.rados.rados.0.trial153.stdout:962: done (3 left) 2026-02-20T12:03:18.600 INFO:tasks.rados.rados.0.trial153.stdout:963: write (excl) oid 15 current snap is 116 2026-02-20T12:03:18.600 INFO:tasks.rados.rados.0.trial153.stdout:963: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2026-02-20T12:03:18.605 INFO:tasks.rados.rados.0.trial153.stdout:963: writing trial15342161-15 from 716097 to 1255909 tid 1 2026-02-20T12:03:18.610 INFO:tasks.rados.rados.0.trial153.stdout:963: writing trial15342161-15 from 1945607 to 2740773 tid 2 2026-02-20T12:03:18.615 INFO:tasks.rados.rados.0.trial153.stdout:963: writing trial15342161-15 from 3535053 to 3916097 tid 3 2026-02-20T12:03:18.615 INFO:tasks.rados.rados.0.trial153.stdout:964: read oid 5 snap -1 2026-02-20T12:03:18.615 INFO:tasks.rados.rados.0.trial153.stdout:964: expect (ObjNum 456 snap 112 seq_num 456) 2026-02-20T12:03:18.616 INFO:tasks.rados.rados.0.trial153.stdout:965: delete oid 7 current snap is 116 2026-02-20T12:03:18.620 INFO:tasks.rados.rados.0.trial153.stdout:965: done (5 left) 2026-02-20T12:03:18.620 INFO:tasks.rados.rados.0.trial153.stdout:966: read oid 27 snap -1 2026-02-20T12:03:18.620 INFO:tasks.rados.rados.0.trial153.stdout:966: expect deleted 2026-02-20T12:03:18.620 INFO:tasks.rados.rados.0.trial153.stdout:967: write oid 22 current snap is 116 2026-02-20T12:03:18.620 INFO:tasks.rados.rados.0.trial153.stdout:967: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2026-02-20T12:03:18.624 INFO:tasks.rados.rados.0.trial153.stdout:967: writing trial15342161-22 from 749711 to 1156374 tid 1 2026-02-20T12:03:18.629 INFO:tasks.rados.rados.0.trial153.stdout:967: writing trial15342161-22 from 1778924 to 2461406 tid 2 2026-02-20T12:03:18.635 INFO:tasks.rados.rados.0.trial153.stdout:967: writing trial15342161-22 from 3189899 to 3905914 tid 3 2026-02-20T12:03:18.635 INFO:tasks.rados.rados.0.trial153.stdout:968: read oid 29 snap -1 2026-02-20T12:03:18.635 INFO:tasks.rados.rados.0.trial153.stdout:968: expect deleted 2026-02-20T12:03:18.636 INFO:tasks.rados.rados.0.trial153.stdout:969: delete oid 49 current snap is 116 2026-02-20T12:03:18.636 INFO:tasks.rados.rados.0.trial153.stdout:963: finishing write tid 1 to trial15342161-15 2026-02-20T12:03:18.636 INFO:tasks.rados.rados.0.trial153.stdout:959: finishing write tid 1 to trial15342161-32 2026-02-20T12:03:18.645 INFO:tasks.rados.rados.0.trial153.stdout:964: expect (ObjNum 456 snap 112 seq_num 456) 2026-02-20T12:03:18.645 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:18.754 INFO:tasks.rados.rados.0.trial153.stdout:961: expect (ObjNum 332 snap 80 seq_num 332) 2026-02-20T12:03:18.754 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:18.804 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:18.881 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2439325 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:963: finishing write tid 2 to trial15342161-15 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:959: finishing write tid 2 to trial15342161-32 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:959: finishing write tid 3 to trial15342161-32 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:963: finishing write tid 3 to trial15342161-15 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:961: done (8 left) 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:964: done (7 left) 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:966: done (6 left) 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:968: done (5 left) 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:969: done (4 left) 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:970: snap_create 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:959: finishing write tid 4 to trial15342161-32 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:959: finishing write tid 5 to trial15342161-32 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:959: oid 32 updating version 0 to 145 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:959: oid 32 version 145 is already newer than 145 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:959: oid 32 version 145 is already newer than 145 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:959: oid 32 version 145 is already newer than 145 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 32 v 145 (ObjNum 468 snap 116 seq_num 468) dirty exists 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:959: left oid 32 (ObjNum 468 snap 116 seq_num 468) 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:963: finishing write tid 4 to trial15342161-15 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:963: finishing write tid 5 to trial15342161-15 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:963: oid 15 updating version 0 to 284 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:963: oid 15 version 284 is already newer than 281 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:963: oid 15 version 284 is already newer than 284 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:963: oid 15 version 284 is already newer than 284 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 15 v 284 (ObjNum 471 snap 116 seq_num 471) dirty exists 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:963: left oid 15 (ObjNum 471 snap 116 seq_num 471) 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:960: finishing write tid 1 to trial15342161-28 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:960: finishing write tid 2 to trial15342161-28 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:960: finishing write tid 3 to trial15342161-28 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:960: finishing write tid 4 to trial15342161-28 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:960: finishing write tid 5 to trial15342161-28 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:960: oid 28 updating version 0 to 192 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:960: oid 28 version 192 is already newer than 192 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:960: oid 28 version 192 is already newer than 192 2026-02-20T12:03:18.900 INFO:tasks.rados.rados.0.trial153.stdout:960: oid 28 version 192 is already newer than 192 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 28 v 192 (ObjNum 469 snap 116 seq_num 469) dirty exists 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:960: left oid 28 (ObjNum 469 snap 116 seq_num 469) 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:967: finishing write tid 1 to trial15342161-22 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:967: finishing write tid 2 to trial15342161-22 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:967: finishing write tid 3 to trial15342161-22 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:967: finishing write tid 4 to trial15342161-22 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:967: finishing write tid 5 to trial15342161-22 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:967: oid 22 updating version 0 to 197 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:967: oid 22 version 197 is already newer than 197 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:967: oid 22 version 197 is already newer than 197 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:967: oid 22 version 197 is already newer than 197 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 22 v 197 (ObjNum 473 snap 116 seq_num 473) dirty exists 2026-02-20T12:03:18.901 INFO:tasks.rados.rados.0.trial153.stdout:967: left oid 22 (ObjNum 473 snap 116 seq_num 473) 2026-02-20T12:03:19.248 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:19.326 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:19.320+0000 7f7489b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:19.328 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:19.324+0000 7f7489b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:19.355 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:19.370 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:19.371 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:19.570 INFO:tasks.rados.rados.0.trial153.stdout:959: done (4 left) 2026-02-20T12:03:19.570 INFO:tasks.rados.rados.0.trial153.stdout:960: done (3 left) 2026-02-20T12:03:19.570 INFO:tasks.rados.rados.0.trial153.stdout:963: done (2 left) 2026-02-20T12:03:19.570 INFO:tasks.rados.rados.0.trial153.stdout:967: done (1 left) 2026-02-20T12:03:19.570 INFO:tasks.rados.rados.0.trial153.stdout:970: done (0 left) 2026-02-20T12:03:19.570 INFO:tasks.rados.rados.0.trial153.stdout:971: write oid 34 current snap is 117 2026-02-20T12:03:19.570 INFO:tasks.rados.rados.0.trial153.stdout:971: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2026-02-20T12:03:19.574 INFO:tasks.rados.rados.0.trial153.stdout:971: writing trial15342161-34 from 783325 to 1340486 tid 1 2026-02-20T12:03:19.579 INFO:tasks.rados.rados.0.trial153.stdout:971: writing trial15342161-34 from 1779535 to 2465686 tid 2 2026-02-20T12:03:19.584 INFO:tasks.rados.rados.0.trial153.stdout:971: writing trial15342161-34 from 3244745 to 3699657 tid 3 2026-02-20T12:03:19.584 INFO:tasks.rados.rados.0.trial153.stdout:972: snap_remove snap 113 2026-02-20T12:03:20.571 INFO:tasks.rados.rados.0.trial153.stdout:972: done (1 left) 2026-02-20T12:03:20.571 INFO:tasks.rados.rados.0.trial153.stdout:973: delete oid 9 current snap is 117 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:971: finishing write tid 1 to trial15342161-34 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:971: finishing write tid 2 to trial15342161-34 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:971: finishing write tid 3 to trial15342161-34 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:971: finishing write tid 4 to trial15342161-34 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:971: finishing write tid 5 to trial15342161-34 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:971: oid 34 updating version 0 to 174 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:971: oid 34 version 174 is already newer than 174 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:971: oid 34 version 174 is already newer than 174 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:971: oid 34 version 174 is already newer than 174 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 174 (ObjNum 475 snap 117 seq_num 475) dirty exists 2026-02-20T12:03:20.572 INFO:tasks.rados.rados.0.trial153.stdout:971: left oid 34 (ObjNum 475 snap 117 seq_num 475) 2026-02-20T12:03:20.635 INFO:tasks.rados.rados.0.trial153.stdout:971: done (1 left) 2026-02-20T12:03:20.635 INFO:tasks.rados.rados.0.trial153.stdout:973: done (0 left) 2026-02-20T12:03:20.635 INFO:tasks.rados.rados.0.trial153.stdout:974: snap_create 2026-02-20T12:03:21.046 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:21.120 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:21.123 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:21.372 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:21.450 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:21.444+0000 7f564e0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:21.453 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:21.448+0000 7f564e0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:21.480 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:21.490 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~248s 2026-02-20T12:03:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:21.497 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:21.575 INFO:tasks.rados.rados.0.trial153.stdout:974: done (0 left) 2026-02-20T12:03:21.576 INFO:tasks.rados.rados.0.trial153.stdout:975: snap_remove snap 117 2026-02-20T12:03:22.578 INFO:tasks.rados.rados.0.trial153.stdout:975: done (0 left) 2026-02-20T12:03:22.578 INFO:tasks.rados.rados.0.trial153.stdout:976: write (excl) oid 43 current snap is 118 2026-02-20T12:03:22.579 INFO:tasks.rados.rados.0.trial153.stdout:976: seq_num 477 ranges {416939=707659,1896499=120440} 2026-02-20T12:03:22.582 INFO:tasks.rados.rados.0.trial153.stdout:976: writing trial15342161-43 from 416939 to 1124598 tid 1 2026-02-20T12:03:22.586 INFO:tasks.rados.rados.0.trial153.stdout:976: writing trial15342161-43 from 1896499 to 2016939 tid 2 2026-02-20T12:03:22.586 INFO:tasks.rados.rados.0.trial153.stdout:977: snap_remove snap 115 2026-02-20T12:03:23.498 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:23.575 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:23.570+0000 7f733a99c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:23.577 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:23.573+0000 7f733a99c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:23.584 INFO:tasks.rados.rados.0.trial153.stdout:977: done (1 left) 2026-02-20T12:03:23.585 INFO:tasks.rados.rados.0.trial153.stdout:978: delete oid 28 current snap is 118 2026-02-20T12:03:23.585 INFO:tasks.rados.rados.0.trial153.stdout:976: finishing write tid 1 to trial15342161-43 2026-02-20T12:03:23.585 INFO:tasks.rados.rados.0.trial153.stdout:976: finishing write tid 2 to trial15342161-43 2026-02-20T12:03:23.585 INFO:tasks.rados.rados.0.trial153.stdout:976: finishing write tid 3 to trial15342161-43 2026-02-20T12:03:23.585 INFO:tasks.rados.rados.0.trial153.stdout:976: finishing write tid 4 to trial15342161-43 2026-02-20T12:03:23.585 INFO:tasks.rados.rados.0.trial153.stdout:976: oid 43 updating version 0 to 85 2026-02-20T12:03:23.585 INFO:tasks.rados.rados.0.trial153.stdout:976: oid 43 version 85 is already newer than 85 2026-02-20T12:03:23.585 INFO:tasks.rados.rados.0.trial153.stdout:976: oid 43 version 85 is already newer than 83 2026-02-20T12:03:23.585 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 43 v 85 (ObjNum 477 snap 118 seq_num 477) dirty exists 2026-02-20T12:03:23.585 INFO:tasks.rados.rados.0.trial153.stdout:976: left oid 43 (ObjNum 477 snap 118 seq_num 477) 2026-02-20T12:03:23.625 INFO:tasks.rados.rados.0.trial153.stdout:976: done (1 left) 2026-02-20T12:03:23.625 INFO:tasks.rados.rados.0.trial153.stdout:978: done (0 left) 2026-02-20T12:03:23.625 INFO:tasks.rados.rados.0.trial153.stdout:979: delete oid 45 current snap is 118 2026-02-20T12:03:23.634 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:23.653 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:23.662 INFO:tasks.rados.rados.0.trial153.stdout:979: done (0 left) 2026-02-20T12:03:23.662 INFO:tasks.rados.rados.0.trial153.stdout:980: read oid 27 snap -1 2026-02-20T12:03:23.662 INFO:tasks.rados.rados.0.trial153.stdout:980: expect deleted 2026-02-20T12:03:23.662 INFO:tasks.rados.rados.0.trial153.stdout:981: snap_remove snap 116 2026-02-20T12:03:24.587 INFO:tasks.rados.rados.0.trial153.stdout:981: done (1 left) 2026-02-20T12:03:24.587 INFO:tasks.rados.rados.0.trial153.stdout:982: write (excl) oid 32 current snap is 118 2026-02-20T12:03:24.587 INFO:tasks.rados.rados.0.trial153.stdout:982: seq_num 480 ranges {467360=649759,1671945=395415} 2026-02-20T12:03:24.591 INFO:tasks.rados.rados.0.trial153.stdout:982: writing trial15342161-32 from 467360 to 1117119 tid 1 2026-02-20T12:03:24.595 INFO:tasks.rados.rados.0.trial153.stdout:982: writing trial15342161-32 from 1671945 to 2067360 tid 2 2026-02-20T12:03:24.595 INFO:tasks.rados.rados.0.trial153.stdout:983: snap_create 2026-02-20T12:03:24.616 INFO:tasks.rados.rados.0.trial153.stdout:982: finishing write tid 1 to trial15342161-32 2026-02-20T12:03:24.628 INFO:tasks.rados.rados.0.trial153.stdout:982: finishing write tid 2 to trial15342161-32 2026-02-20T12:03:24.635 INFO:tasks.rados.rados.0.trial153.stdout:982: finishing write tid 3 to trial15342161-32 2026-02-20T12:03:24.637 INFO:tasks.rados.rados.0.trial153.stdout:982: finishing write tid 4 to trial15342161-32 2026-02-20T12:03:24.637 INFO:tasks.rados.rados.0.trial153.stdout:982: oid 32 updating version 0 to 148 2026-02-20T12:03:24.637 INFO:tasks.rados.rados.0.trial153.stdout:982: oid 32 version 148 is already newer than 146 2026-02-20T12:03:24.637 INFO:tasks.rados.rados.0.trial153.stdout:982: oid 32 version 148 is already newer than 148 2026-02-20T12:03:24.637 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 32 v 148 (ObjNum 480 snap 118 seq_num 480) dirty exists 2026-02-20T12:03:24.637 INFO:tasks.rados.rados.0.trial153.stdout:982: left oid 32 (ObjNum 480 snap 118 seq_num 480) 2026-02-20T12:03:25.588 INFO:tasks.rados.rados.0.trial153.stdout:980: done (2 left) 2026-02-20T12:03:25.588 INFO:tasks.rados.rados.0.trial153.stdout:982: done (1 left) 2026-02-20T12:03:25.588 INFO:tasks.rados.rados.0.trial153.stdout:983: done (0 left) 2026-02-20T12:03:25.588 INFO:tasks.rados.rados.0.trial153.stdout:984: delete oid 40 current snap is 119 2026-02-20T12:03:25.602 INFO:tasks.rados.rados.0.trial153.stdout:984: done (0 left) 2026-02-20T12:03:25.602 INFO:tasks.rados.rados.0.trial153.stdout:985: write (excl) oid 27 current snap is 119 2026-02-20T12:03:25.602 INFO:tasks.rados.rados.0.trial153.stdout:985: seq_num 482 ranges {500974=400257,1388909=557169} 2026-02-20T12:03:25.605 INFO:tasks.rados.rados.0.trial153.stdout:985: writing trial15342161-27 from 500974 to 901231 tid 1 2026-02-20T12:03:25.609 INFO:tasks.rados.rados.0.trial153.stdout:985: writing trial15342161-27 from 1388909 to 1946078 tid 2 2026-02-20T12:03:25.609 INFO:tasks.rados.rados.0.trial153.stdout:986: delete oid 49 current snap is 119 2026-02-20T12:03:25.623 INFO:tasks.rados.rados.0.trial153.stdout:985: finishing write tid 1 to trial15342161-27 2026-02-20T12:03:25.644 INFO:tasks.rados.rados.0.trial153.stdout:985: finishing write tid 2 to trial15342161-27 2026-02-20T12:03:25.644 INFO:tasks.rados.rados.0.trial153.stdout:986: done (1 left) 2026-02-20T12:03:25.645 INFO:tasks.rados.rados.0.trial153.stdout:987: delete oid 21 current snap is 119 2026-02-20T12:03:25.648 INFO:tasks.rados.rados.0.trial153.stdout:987: done (1 left) 2026-02-20T12:03:25.648 INFO:tasks.rados.rados.0.trial153.stdout:988: snap_create 2026-02-20T12:03:25.653 INFO:tasks.rados.rados.0.trial153.stdout:985: finishing write tid 3 to trial15342161-27 2026-02-20T12:03:25.653 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:25.656 INFO:tasks.rados.rados.0.trial153.stdout:985: finishing write tid 4 to trial15342161-27 2026-02-20T12:03:25.656 INFO:tasks.rados.rados.0.trial153.stdout:985: oid 27 updating version 0 to 77 2026-02-20T12:03:25.656 INFO:tasks.rados.rados.0.trial153.stdout:985: oid 27 version 77 is already newer than 77 2026-02-20T12:03:25.656 INFO:tasks.rados.rados.0.trial153.stdout:985: oid 27 version 77 is already newer than 75 2026-02-20T12:03:25.656 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 27 v 77 (ObjNum 482 snap 119 seq_num 482) dirty exists 2026-02-20T12:03:25.656 INFO:tasks.rados.rados.0.trial153.stdout:985: left oid 27 (ObjNum 482 snap 119 seq_num 482) 2026-02-20T12:03:25.727 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:25.722+0000 7fbee710b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:25.729 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:25.725+0000 7fbee710b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:25.757 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:25.777 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:25.777 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:26.124 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:26.197 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:26.199 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:26.200 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:26.530 INFO:tasks.rados.rados.0.trial153.stdout:985: done (1 left) 2026-02-20T12:03:26.530 INFO:tasks.rados.rados.0.trial153.stdout:988: done (0 left) 2026-02-20T12:03:26.531 INFO:tasks.rados.rados.0.trial153.stdout:989: snap_create 2026-02-20T12:03:26.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~253s 2026-02-20T12:03:27.535 INFO:tasks.rados.rados.0.trial153.stdout:989: done (0 left) 2026-02-20T12:03:27.535 INFO:tasks.rados.rados.0.trial153.stdout:990: read oid 20 snap -1 2026-02-20T12:03:27.535 INFO:tasks.rados.rados.0.trial153.stdout:990: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:03:27.535 INFO:tasks.rados.rados.0.trial153.stdout:991: read oid 29 snap -1 2026-02-20T12:03:27.535 INFO:tasks.rados.rados.0.trial153.stdout:991: expect deleted 2026-02-20T12:03:27.535 INFO:tasks.rados.rados.0.trial153.stdout:992: delete oid 24 current snap is 121 2026-02-20T12:03:27.570 INFO:tasks.rados.rados.0.trial153.stdout:990: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:03:27.667 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:27.720 INFO:tasks.rados.rados.0.trial153.stdout:990: done (2 left) 2026-02-20T12:03:27.720 INFO:tasks.rados.rados.0.trial153.stdout:991: done (1 left) 2026-02-20T12:03:27.720 INFO:tasks.rados.rados.0.trial153.stdout:992: done (0 left) 2026-02-20T12:03:27.720 INFO:tasks.rados.rados.0.trial153.stdout:993: snap_create 2026-02-20T12:03:27.778 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:27.853 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:27.848+0000 7fd8cb251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:27.856 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:27.852+0000 7fd8cb251640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:27.884 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:27.900 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:27.900 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:28.594 INFO:tasks.rados.rados.0.trial153.stdout:993: done (0 left) 2026-02-20T12:03:28.594 INFO:tasks.rados.rados.0.trial153.stdout:994: delete oid 46 current snap is 122 2026-02-20T12:03:28.607 INFO:tasks.rados.rados.0.trial153.stdout:994: done (0 left) 2026-02-20T12:03:28.607 INFO:tasks.rados.rados.0.trial153.stdout:995: delete oid 16 current snap is 122 2026-02-20T12:03:28.636 INFO:tasks.rados.rados.0.trial153.stdout:995: done (0 left) 2026-02-20T12:03:28.636 INFO:tasks.rados.rados.0.trial153.stdout:996: write (excl) oid 24 current snap is 122 2026-02-20T12:03:28.636 INFO:tasks.rados.rados.0.trial153.stdout:996: seq_num 488 ranges {601816=568104,1623448=451823} 2026-02-20T12:03:28.642 INFO:tasks.rados.rados.0.trial153.stdout:996: writing trial15342161-24 from 601816 to 1169920 tid 1 2026-02-20T12:03:28.646 INFO:tasks.rados.rados.0.trial153.stdout:996: writing trial15342161-24 from 1623448 to 2075271 tid 2 2026-02-20T12:03:28.646 INFO:tasks.rados.rados.0.trial153.stdout:997: read oid 40 snap -1 2026-02-20T12:03:28.646 INFO:tasks.rados.rados.0.trial153.stdout:997: expect deleted 2026-02-20T12:03:28.646 INFO:tasks.rados.rados.0.trial153.stdout:998: write oid 6 current snap is 122 2026-02-20T12:03:28.646 INFO:tasks.rados.rados.0.trial153.stdout:998: seq_num 489 ranges {618623=643353,1681930=536693} 2026-02-20T12:03:28.650 INFO:tasks.rados.rados.0.trial153.stdout:998: writing trial15342161-6 from 618623 to 1261976 tid 1 2026-02-20T12:03:28.654 INFO:tasks.rados.rados.0.trial153.stdout:998: writing trial15342161-6 from 1681930 to 2218623 tid 2 2026-02-20T12:03:28.654 INFO:tasks.rados.rados.0.trial153.stdout:999: read oid 8 snap 120 2026-02-20T12:03:28.654 INFO:tasks.rados.rados.0.trial153.stdout:999: expect (ObjNum 451 snap 112 seq_num 451) 2026-02-20T12:03:28.654 INFO:tasks.rados.rados.0.trial153.stdout:1000: delete oid 7 current snap is 122 2026-02-20T12:03:28.657 INFO:tasks.rados.rados.0.trial153.stdout:997: done (4 left) 2026-02-20T12:03:28.657 INFO:tasks.rados.rados.0.trial153.stdout:1000: done (3 left) 2026-02-20T12:03:28.657 INFO:tasks.rados.rados.0.trial153.stdout:1001: write (excl) oid 40 current snap is 122 2026-02-20T12:03:28.657 INFO:tasks.rados.rados.0.trial153.stdout:1001: seq_num 491 ranges {652237=793851,2198894=53343} 2026-02-20T12:03:28.663 INFO:tasks.rados.rados.0.trial153.stdout:1001: writing trial15342161-40 from 652237 to 1446088 tid 1 2026-02-20T12:03:28.666 INFO:tasks.rados.rados.0.trial153.stdout:1001: writing trial15342161-40 from 2198894 to 2252237 tid 2 2026-02-20T12:03:28.666 INFO:tasks.rados.rados.0.trial153.stdout:1002: read oid 14 snap -1 2026-02-20T12:03:28.666 INFO:tasks.rados.rados.0.trial153.stdout:1002: expect (ObjNum 466 snap 115 seq_num 466) 2026-02-20T12:03:28.666 INFO:tasks.rados.rados.0.trial153.stdout:1003: read oid 32 snap -1 2026-02-20T12:03:28.666 INFO:tasks.rados.rados.0.trial153.stdout:1003: expect (ObjNum 480 snap 118 seq_num 480) 2026-02-20T12:03:28.666 INFO:tasks.rados.rados.0.trial153.stdout:1004: delete oid 48 current snap is 122 2026-02-20T12:03:28.668 INFO:tasks.rados.rados.0.trial153.stdout:996: finishing write tid 1 to trial15342161-24 2026-02-20T12:03:28.680 INFO:tasks.rados.rados.0.trial153.stdout:998: finishing write tid 1 to trial15342161-6 2026-02-20T12:03:28.691 INFO:tasks.rados.rados.0.trial153.stdout:996: finishing write tid 2 to trial15342161-24 2026-02-20T12:03:28.694 INFO:tasks.rados.rados.0.trial153.stdout:1003: expect (ObjNum 480 snap 118 seq_num 480) 2026-02-20T12:03:28.718 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:28.774 INFO:tasks.rados.rados.0.trial153.stdout:998: finishing write tid 2 to trial15342161-6 2026-02-20T12:03:28.774 INFO:tasks.rados.rados.0.trial153.stdout:996: finishing write tid 3 to trial15342161-24 2026-02-20T12:03:28.774 INFO:tasks.rados.rados.0.trial153.stdout:996: finishing write tid 4 to trial15342161-24 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:996: oid 24 updating version 0 to 31 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:996: oid 24 updating version 31 to 33 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:996: oid 24 version 33 is already newer than 33 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 24 v 33 (ObjNum 488 snap 122 seq_num 488) dirty exists 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:996: left oid 24 (ObjNum 488 snap 122 seq_num 488) 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:998: finishing write tid 3 to trial15342161-6 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:998: finishing write tid 4 to trial15342161-6 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:998: oid 6 updating version 0 to 291 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:998: oid 6 version 291 is already newer than 291 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:998: oid 6 version 291 is already newer than 291 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 6 v 291 (ObjNum 489 snap 122 seq_num 489) dirty exists 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:998: left oid 6 (ObjNum 489 snap 122 seq_num 489) 2026-02-20T12:03:28.775 INFO:tasks.rados.rados.0.trial153.stdout:999: expect (ObjNum 451 snap 112 seq_num 451) 2026-02-20T12:03:28.890 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:28.956 INFO:tasks.rados.rados.0.trial153.stdout:1001: finishing write tid 1 to trial15342161-40 2026-02-20T12:03:28.957 INFO:tasks.rados.rados.0.trial153.stdout:1001: finishing write tid 2 to trial15342161-40 2026-02-20T12:03:28.957 INFO:tasks.rados.rados.0.trial153.stdout:996: done (6 left) 2026-02-20T12:03:28.957 INFO:tasks.rados.rados.0.trial153.stdout:998: done (5 left) 2026-02-20T12:03:28.957 INFO:tasks.rados.rados.0.trial153.stdout:999: done (4 left) 2026-02-20T12:03:28.957 INFO:tasks.rados.rados.0.trial153.stdout:1003: done (3 left) 2026-02-20T12:03:28.957 INFO:tasks.rados.rados.0.trial153.stdout:1004: done (2 left) 2026-02-20T12:03:28.957 INFO:tasks.rados.rados.0.trial153.stdout:1005: write (excl) oid 42 current snap is 122 2026-02-20T12:03:28.957 INFO:tasks.rados.rados.0.trial153.stdout:1001: finishing write tid 3 to trial15342161-40 2026-02-20T12:03:28.958 INFO:tasks.rados.rados.0.trial153.stdout:1005: seq_num 493 ranges {685851=544349,1799505=486346} 2026-02-20T12:03:28.960 INFO:tasks.rados.rados.0.trial153.stdout:1005: writing trial15342161-42 from 685851 to 1230200 tid 1 2026-02-20T12:03:28.965 INFO:tasks.rados.rados.0.trial153.stdout:1005: writing trial15342161-42 from 1799505 to 2285851 tid 2 2026-02-20T12:03:28.965 INFO:tasks.rados.rados.0.trial153.stdout:1006: read oid 4 snap 121 2026-02-20T12:03:28.965 INFO:tasks.rados.rados.0.trial153.stdout:1006: expect deleted 2026-02-20T12:03:28.965 INFO:tasks.rados.rados.0.trial153.stdout:1007: write oid 21 current snap is 122 2026-02-20T12:03:28.965 INFO:tasks.rados.rados.0.trial153.stdout:1001: finishing write tid 4 to trial15342161-40 2026-02-20T12:03:28.965 INFO:tasks.rados.rados.0.trial153.stdout:1001: oid 40 updating version 0 to 202 2026-02-20T12:03:28.965 INFO:tasks.rados.rados.0.trial153.stdout:1001: oid 40 version 202 is already newer than 202 2026-02-20T12:03:28.966 INFO:tasks.rados.rados.0.trial153.stdout:1001: oid 40 version 202 is already newer than 200 2026-02-20T12:03:28.966 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 40 v 202 (ObjNum 491 snap 122 seq_num 491) dirty exists 2026-02-20T12:03:28.966 INFO:tasks.rados.rados.0.trial153.stdout:1001: left oid 40 (ObjNum 491 snap 122 seq_num 491) 2026-02-20T12:03:28.966 INFO:tasks.rados.rados.0.trial153.stdout:1002: expect (ObjNum 466 snap 115 seq_num 466) 2026-02-20T12:03:29.145 INFO:tasks.rados.rados.0.trial153.stdout:1007: seq_num 494 ranges {702658=619598,1857987=444671} 2026-02-20T12:03:29.150 INFO:tasks.rados.rados.0.trial153.stdout:1007: writing trial15342161-21 from 702658 to 1322256 tid 1 2026-02-20T12:03:29.154 INFO:tasks.rados.rados.0.trial153.stdout:1007: writing trial15342161-21 from 1857987 to 2302658 tid 2 2026-02-20T12:03:29.154 INFO:tasks.rados.rados.0.trial153.stdout:1001: done (4 left) 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:1002: done (3 left) 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:1008: snap_create 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:1005: finishing write tid 1 to trial15342161-42 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:1005: finishing write tid 2 to trial15342161-42 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:1005: finishing write tid 3 to trial15342161-42 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:1005: finishing write tid 4 to trial15342161-42 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:1005: oid 42 updating version 0 to 81 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:1005: oid 42 version 81 is already newer than 81 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:1005: oid 42 version 81 is already newer than 79 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 81 (ObjNum 493 snap 122 seq_num 493) dirty exists 2026-02-20T12:03:29.155 INFO:tasks.rados.rados.0.trial153.stdout:1005: left oid 42 (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:29.170 INFO:tasks.rados.rados.0.trial153.stdout:1007: finishing write tid 1 to trial15342161-21 2026-02-20T12:03:29.176 INFO:tasks.rados.rados.0.trial153.stdout:1007: finishing write tid 2 to trial15342161-21 2026-02-20T12:03:29.181 INFO:tasks.rados.rados.0.trial153.stdout:1007: finishing write tid 3 to trial15342161-21 2026-02-20T12:03:29.183 INFO:tasks.rados.rados.0.trial153.stdout:1007: finishing write tid 4 to trial15342161-21 2026-02-20T12:03:29.183 INFO:tasks.rados.rados.0.trial153.stdout:1007: oid 21 updating version 0 to 179 2026-02-20T12:03:29.183 INFO:tasks.rados.rados.0.trial153.stdout:1007: oid 21 version 179 is already newer than 179 2026-02-20T12:03:29.183 INFO:tasks.rados.rados.0.trial153.stdout:1007: oid 21 version 179 is already newer than 179 2026-02-20T12:03:29.183 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 21 v 179 (ObjNum 494 snap 122 seq_num 494) dirty exists 2026-02-20T12:03:29.183 INFO:tasks.rados.rados.0.trial153.stdout:1007: left oid 21 (ObjNum 494 snap 122 seq_num 494) 2026-02-20T12:03:29.598 INFO:tasks.rados.rados.0.trial153.stdout:1005: done (3 left) 2026-02-20T12:03:29.598 INFO:tasks.rados.rados.0.trial153.stdout:1006: done (2 left) 2026-02-20T12:03:29.598 INFO:tasks.rados.rados.0.trial153.stdout:1007: done (1 left) 2026-02-20T12:03:29.598 INFO:tasks.rados.rados.0.trial153.stdout:1008: done (0 left) 2026-02-20T12:03:29.598 INFO:tasks.rados.rados.0.trial153.stdout:1009: read oid 30 snap -1 2026-02-20T12:03:29.598 INFO:tasks.rados.rados.0.trial153.stdout:1009: expect (ObjNum 464 snap 114 seq_num 464) 2026-02-20T12:03:29.598 INFO:tasks.rados.rados.0.trial153.stdout:1010: delete oid 15 current snap is 123 2026-02-20T12:03:29.649 INFO:tasks.rados.rados.0.trial153.stdout:1010: done (1 left) 2026-02-20T12:03:29.649 INFO:tasks.rados.rados.0.trial153.stdout:1011: delete oid 41 current snap is 123 2026-02-20T12:03:29.651 INFO:tasks.rados.rados.0.trial153.stdout:1009: expect (ObjNum 464 snap 114 seq_num 464) 2026-02-20T12:03:29.686 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 3523700 2026-02-20T12:03:29.692 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:29.796 INFO:tasks.rados.rados.0.trial153.stdout:1009: done (1 left) 2026-02-20T12:03:29.796 INFO:tasks.rados.rados.0.trial153.stdout:1011: done (0 left) 2026-02-20T12:03:29.796 INFO:tasks.rados.rados.0.trial153.stdout:1012: read oid 36 snap -1 2026-02-20T12:03:29.796 INFO:tasks.rados.rados.0.trial153.stdout:1012: expect deleted 2026-02-20T12:03:29.796 INFO:tasks.rados.rados.0.trial153.stdout:1013: delete oid 16 current snap is 123 2026-02-20T12:03:29.798 INFO:tasks.rados.rados.0.trial153.stdout:1013: done (1 left) 2026-02-20T12:03:29.798 INFO:tasks.rados.rados.0.trial153.stdout:1014: read oid 49 snap 118 2026-02-20T12:03:29.798 INFO:tasks.rados.rados.0.trial153.stdout:1014: expect deleted 2026-02-20T12:03:29.798 INFO:tasks.rados.rados.0.trial153.stdout:1015: read oid 39 snap -1 2026-02-20T12:03:29.798 INFO:tasks.rados.rados.0.trial153.stdout:1015: expect (ObjNum 407 snap 97 seq_num 407) 2026-02-20T12:03:29.799 INFO:tasks.rados.rados.0.trial153.stdout:1016: write (excl) oid 9 current snap is 123 2026-02-20T12:03:29.799 INFO:tasks.rados.rados.0.trial153.stdout:1016: seq_num 498 ranges {769886=636947,2091915=277971} 2026-02-20T12:03:29.803 INFO:tasks.rados.rados.0.trial153.stdout:1016: writing trial15342161-9 from 769886 to 1406833 tid 1 2026-02-20T12:03:29.807 INFO:tasks.rados.rados.0.trial153.stdout:1016: writing trial15342161-9 from 2091915 to 2369886 tid 2 2026-02-20T12:03:29.807 INFO:tasks.rados.rados.0.trial153.stdout:1017: read oid 43 snap -1 2026-02-20T12:03:29.807 INFO:tasks.rados.rados.0.trial153.stdout:1017: expect (ObjNum 477 snap 118 seq_num 477) 2026-02-20T12:03:29.808 INFO:tasks.rados.rados.0.trial153.stdout:1018: delete oid 34 current snap is 123 2026-02-20T12:03:29.815 INFO:tasks.rados.rados.0.trial153.stdout:1015: expect (ObjNum 407 snap 97 seq_num 407) 2026-02-20T12:03:29.836 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 866547 2026-02-20T12:03:29.852 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:29.876 INFO:tasks.rados.rados.0.trial153.stdout:1016: finishing write tid 1 to trial15342161-9 2026-02-20T12:03:29.877 INFO:tasks.rados.rados.0.trial153.stdout:1017: expect (ObjNum 477 snap 118 seq_num 477) 2026-02-20T12:03:29.893 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1715788 2026-02-20T12:03:29.900 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:29.907 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:29.977 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:29.972+0000 7f0ce9ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:29.979 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:29.975+0000 7f0ce9ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:29.985 INFO:tasks.rados.rados.0.trial153.stdout:1016: finishing write tid 2 to trial15342161-9 2026-02-20T12:03:29.985 INFO:tasks.rados.rados.0.trial153.stdout:1016: finishing write tid 3 to trial15342161-9 2026-02-20T12:03:29.985 INFO:tasks.rados.rados.0.trial153.stdout:1012: done (5 left) 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1014: done (4 left) 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1015: done (3 left) 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1017: done (2 left) 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1018: done (1 left) 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1019: write oid 19 current snap is 123 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1016: finishing write tid 4 to trial15342161-9 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1016: oid 9 updating version 0 to 290 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1016: oid 9 version 290 is already newer than 290 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1016: oid 9 version 290 is already newer than 288 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 9 v 290 (ObjNum 498 snap 123 seq_num 498) dirty exists 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1016: left oid 9 (ObjNum 498 snap 123 seq_num 498) 2026-02-20T12:03:29.986 INFO:tasks.rados.rados.0.trial153.stdout:1019: seq_num 500 ranges {403500=787445,1692526=473837} 2026-02-20T12:03:29.989 INFO:tasks.rados.rados.0.trial153.stdout:1019: writing trial15342161-19 from 403500 to 1190945 tid 1 2026-02-20T12:03:29.993 INFO:tasks.rados.rados.0.trial153.stdout:1019: writing trial15342161-19 from 1692526 to 2166363 tid 2 2026-02-20T12:03:29.993 INFO:tasks.rados.rados.0.trial153.stdout:1016: done (1 left) 2026-02-20T12:03:29.993 INFO:tasks.rados.rados.0.trial153.stdout:1020: write oid 26 current snap is 123 2026-02-20T12:03:29.993 INFO:tasks.rados.rados.0.trial153.stdout:1020: seq_num 501 ranges {420307=462694,1351008=617495} 2026-02-20T12:03:29.996 INFO:tasks.rados.rados.0.trial153.stdout:1020: writing trial15342161-26 from 420307 to 883001 tid 1 2026-02-20T12:03:30.001 INFO:tasks.rados.rados.0.trial153.stdout:1020: writing trial15342161-26 from 1351008 to 1968503 tid 2 2026-02-20T12:03:30.001 INFO:tasks.rados.rados.0.trial153.stdout:1021: write (excl) oid 15 current snap is 123 2026-02-20T12:03:30.001 INFO:tasks.rados.rados.0.trial153.stdout:1021: seq_num 502 ranges {437114=654296,1525843=477506} 2026-02-20T12:03:30.006 INFO:tasks.rados.rados.0.trial153.stdout:1021: writing trial15342161-15 from 437114 to 1091410 tid 1 2026-02-20T12:03:30.010 INFO:tasks.rados.rados.0.trial153.stdout:1021: writing trial15342161-15 from 1525843 to 2003349 tid 2 2026-02-20T12:03:30.010 INFO:tasks.rados.rados.0.trial153.stdout:1022: snap_remove snap 120 2026-02-20T12:03:30.013 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:30.034 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:30.600 INFO:tasks.rados.rados.0.trial153.stdout:1022: done (3 left) 2026-02-20T12:03:30.600 INFO:tasks.rados.rados.0.trial153.stdout:1023: snap_remove snap 119 2026-02-20T12:03:31.201 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:31.272 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:31.275 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:31.604 INFO:tasks.rados.rados.0.trial153.stdout:1023: done (3 left) 2026-02-20T12:03:31.605 INFO:tasks.rados.rados.0.trial153.stdout:1024: snap_remove snap 118 2026-02-20T12:03:32.035 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:32.113 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:32.108+0000 7fbd1addc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:32.116 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:32.111+0000 7fbd1addc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:32.143 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:32.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:32.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~259s 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1024: done (3 left) 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1025: delete oid 25 current snap is 123 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1020: finishing write tid 1 to trial15342161-26 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1019: finishing write tid 1 to trial15342161-19 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1020: finishing write tid 2 to trial15342161-26 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1019: finishing write tid 2 to trial15342161-19 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1020: finishing write tid 3 to trial15342161-26 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1019: finishing write tid 3 to trial15342161-19 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1020: finishing write tid 4 to trial15342161-26 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1020: oid 26 updating version 0 to 219 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1020: oid 26 version 219 is already newer than 219 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1020: oid 26 version 219 is already newer than 219 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 26 v 219 (ObjNum 501 snap 123 seq_num 501) dirty exists 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1020: left oid 26 (ObjNum 501 snap 123 seq_num 501) 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1019: finishing write tid 4 to trial15342161-19 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1019: oid 19 updating version 0 to 297 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1019: oid 19 version 297 is already newer than 297 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1019: oid 19 version 297 is already newer than 297 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 297 (ObjNum 500 snap 123 seq_num 500) dirty exists 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1019: left oid 19 (ObjNum 500 snap 123 seq_num 500) 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1021: finishing write tid 1 to trial15342161-15 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1021: finishing write tid 2 to trial15342161-15 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1021: finishing write tid 3 to trial15342161-15 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1021: finishing write tid 4 to trial15342161-15 2026-02-20T12:03:32.607 INFO:tasks.rados.rados.0.trial153.stdout:1021: oid 15 updating version 0 to 298 2026-02-20T12:03:32.608 INFO:tasks.rados.rados.0.trial153.stdout:1021: oid 15 updating version 298 to 300 2026-02-20T12:03:32.608 INFO:tasks.rados.rados.0.trial153.stdout:1021: oid 15 version 300 is already newer than 300 2026-02-20T12:03:32.608 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 15 v 300 (ObjNum 502 snap 123 seq_num 502) dirty exists 2026-02-20T12:03:32.608 INFO:tasks.rados.rados.0.trial153.stdout:1021: left oid 15 (ObjNum 502 snap 123 seq_num 502) 2026-02-20T12:03:32.656 INFO:tasks.rados.rados.0.trial153.stdout:1019: done (3 left) 2026-02-20T12:03:32.656 INFO:tasks.rados.rados.0.trial153.stdout:1020: done (2 left) 2026-02-20T12:03:32.656 INFO:tasks.rados.rados.0.trial153.stdout:1021: done (1 left) 2026-02-20T12:03:32.656 INFO:tasks.rados.rados.0.trial153.stdout:1025: done (0 left) 2026-02-20T12:03:32.656 INFO:tasks.rados.rados.0.trial153.stdout:1026: read oid 29 snap 121 2026-02-20T12:03:32.656 INFO:tasks.rados.rados.0.trial153.stdout:1026: expect deleted 2026-02-20T12:03:32.656 INFO:tasks.rados.rados.0.trial153.stdout:1027: read oid 47 snap -1 2026-02-20T12:03:32.656 INFO:tasks.rados.rados.0.trial153.stdout:1027: expect (ObjNum 447 snap 112 seq_num 447) 2026-02-20T12:03:32.656 INFO:tasks.rados.rados.0.trial153.stdout:1028: read oid 42 snap -1 2026-02-20T12:03:32.657 INFO:tasks.rados.rados.0.trial153.stdout:1028: expect (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:32.657 INFO:tasks.rados.rados.0.trial153.stdout:1029: snap_create 2026-02-20T12:03:32.676 INFO:tasks.rados.rados.0.trial153.stdout:1027: expect (ObjNum 447 snap 112 seq_num 447) 2026-02-20T12:03:32.705 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2522431 2026-02-20T12:03:32.720 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:32.829 INFO:tasks.rados.rados.0.trial153.stdout:1028: expect (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:32.835 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 796647 2026-02-20T12:03:32.894 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1143852 2026-02-20T12:03:33.610 INFO:tasks.rados.rados.0.trial153.stdout:1026: done (3 left) 2026-02-20T12:03:33.610 INFO:tasks.rados.rados.0.trial153.stdout:1027: done (2 left) 2026-02-20T12:03:33.610 INFO:tasks.rados.rados.0.trial153.stdout:1028: done (1 left) 2026-02-20T12:03:33.610 INFO:tasks.rados.rados.0.trial153.stdout:1029: done (0 left) 2026-02-20T12:03:33.610 INFO:tasks.rados.rados.0.trial153.stdout:1030: delete oid 16 current snap is 124 2026-02-20T12:03:33.625 INFO:tasks.rados.rados.0.trial153.stdout:1030: done (0 left) 2026-02-20T12:03:33.625 INFO:tasks.rados.rados.0.trial153.stdout:1031: snap_create 2026-02-20T12:03:34.160 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:34.238 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:34.232+0000 7f6c247a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:34.240 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:34.236+0000 7f6c247a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:34.268 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:34.283 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:34.284 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:34.613 INFO:tasks.rados.rados.0.trial153.stdout:1031: done (0 left) 2026-02-20T12:03:34.614 INFO:tasks.rados.rados.0.trial153.stdout:1032: write (excl) oid 50 current snap is 125 2026-02-20T12:03:34.614 INFO:tasks.rados.rados.0.trial153.stdout:1032: seq_num 505 ranges {487535=480043,1584936=624833} 2026-02-20T12:03:34.617 INFO:tasks.rados.rados.0.trial153.stdout:1032: writing trial15342161-50 from 487535 to 967578 tid 1 2026-02-20T12:03:34.622 INFO:tasks.rados.rados.0.trial153.stdout:1032: writing trial15342161-50 from 1584936 to 2209769 tid 2 2026-02-20T12:03:34.623 INFO:tasks.rados.rados.0.trial153.stdout:1033: read oid 37 snap 122 2026-02-20T12:03:34.623 INFO:tasks.rados.rados.0.trial153.stdout:1033: expect deleted 2026-02-20T12:03:34.623 INFO:tasks.rados.rados.0.trial153.stdout:1034: read oid 42 snap -1 2026-02-20T12:03:34.623 INFO:tasks.rados.rados.0.trial153.stdout:1034: expect (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:34.623 INFO:tasks.rados.rados.0.trial153.stdout:1035: read oid 47 snap -1 2026-02-20T12:03:34.623 INFO:tasks.rados.rados.0.trial153.stdout:1035: expect (ObjNum 447 snap 112 seq_num 447) 2026-02-20T12:03:34.623 INFO:tasks.rados.rados.0.trial153.stdout:1036: write oid 5 current snap is 125 2026-02-20T12:03:34.623 INFO:tasks.rados.rados.0.trial153.stdout:1036: seq_num 506 ranges {504342=555292,1643418=484844} 2026-02-20T12:03:34.626 INFO:tasks.rados.rados.0.trial153.stdout:1036: writing trial15342161-5 from 504342 to 1059634 tid 1 2026-02-20T12:03:34.631 INFO:tasks.rados.rados.0.trial153.stdout:1036: writing trial15342161-5 from 1643418 to 2128262 tid 2 2026-02-20T12:03:34.631 INFO:tasks.rados.rados.0.trial153.stdout:1037: write oid 27 current snap is 125 2026-02-20T12:03:34.631 INFO:tasks.rados.rados.0.trial153.stdout:1037: seq_num 507 ranges {521149=630541,1701900=628502} 2026-02-20T12:03:34.635 INFO:tasks.rados.rados.0.trial153.stdout:1037: writing trial15342161-27 from 521149 to 1151690 tid 1 2026-02-20T12:03:34.640 INFO:tasks.rados.rados.0.trial153.stdout:1037: writing trial15342161-27 from 1701900 to 2330402 tid 2 2026-02-20T12:03:34.640 INFO:tasks.rados.rados.0.trial153.stdout:1038: snap_remove snap 124 2026-02-20T12:03:35.620 INFO:tasks.rados.rados.0.trial153.stdout:1038: done (6 left) 2026-02-20T12:03:35.620 INFO:tasks.rados.rados.0.trial153.stdout:1039: read oid 15 snap -1 2026-02-20T12:03:35.620 INFO:tasks.rados.rados.0.trial153.stdout:1039: expect (ObjNum 502 snap 123 seq_num 502) 2026-02-20T12:03:35.620 INFO:tasks.rados.rados.0.trial153.stdout:1040: snap_create 2026-02-20T12:03:35.620 INFO:tasks.rados.rados.0.trial153.stdout:1035: expect (ObjNum 447 snap 112 seq_num 447) 2026-02-20T12:03:35.620 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:35.676 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:35.730 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:35.786 INFO:tasks.rados.rados.0.trial153.stdout:1034: expect (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:35.836 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2237720 2026-02-20T12:03:35.850 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1590367 2026-02-20T12:03:35.860 INFO:tasks.rados.rados.0.trial153.stdout:1032: finishing write tid 1 to trial15342161-50 2026-02-20T12:03:35.860 INFO:tasks.rados.rados.0.trial153.stdout:1037: finishing write tid 1 to trial15342161-27 2026-02-20T12:03:35.860 INFO:tasks.rados.rados.0.trial153.stdout:1037: finishing write tid 2 to trial15342161-27 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1036: finishing write tid 1 to trial15342161-5 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1037: finishing write tid 3 to trial15342161-27 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1037: finishing write tid 4 to trial15342161-27 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1037: oid 27 updating version 0 to 87 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1037: oid 27 version 87 is already newer than 87 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1037: oid 27 version 87 is already newer than 87 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 27 v 87 (ObjNum 507 snap 125 seq_num 507) dirty exists 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1037: left oid 27 (ObjNum 507 snap 125 seq_num 507) 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1036: finishing write tid 2 to trial15342161-5 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1036: finishing write tid 3 to trial15342161-5 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1036: finishing write tid 4 to trial15342161-5 2026-02-20T12:03:35.861 INFO:tasks.rados.rados.0.trial153.stdout:1036: oid 5 updating version 0 to 102 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:1036: oid 5 version 102 is already newer than 102 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:1036: oid 5 version 102 is already newer than 102 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 5 v 102 (ObjNum 506 snap 125 seq_num 506) dirty exists 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:1036: left oid 5 (ObjNum 506 snap 125 seq_num 506) 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:1032: finishing write tid 2 to trial15342161-50 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:1032: finishing write tid 3 to trial15342161-50 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:1032: finishing write tid 4 to trial15342161-50 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:1032: oid 50 updating version 0 to 310 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:1032: oid 50 updating version 310 to 312 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:1032: oid 50 version 312 is already newer than 312 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 50 v 312 (ObjNum 505 snap 125 seq_num 505) dirty exists 2026-02-20T12:03:35.862 INFO:tasks.rados.rados.0.trial153.stdout:1032: left oid 50 (ObjNum 505 snap 125 seq_num 505) 2026-02-20T12:03:35.863 INFO:tasks.rados.rados.0.trial153.stdout:1039: expect (ObjNum 502 snap 123 seq_num 502) 2026-02-20T12:03:35.880 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:35.901 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:36.276 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:36.285 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:36.348 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:36.351 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:36.363 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:36.358+0000 7fda4f2c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:36.366 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:36.361+0000 7fda4f2c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:36.393 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:36.411 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:36.533 INFO:tasks.rados.rados.0.trial153.stdout:1032: done (7 left) 2026-02-20T12:03:36.533 INFO:tasks.rados.rados.0.trial153.stdout:1033: done (6 left) 2026-02-20T12:03:36.533 INFO:tasks.rados.rados.0.trial153.stdout:1034: done (5 left) 2026-02-20T12:03:36.533 INFO:tasks.rados.rados.0.trial153.stdout:1035: done (4 left) 2026-02-20T12:03:36.533 INFO:tasks.rados.rados.0.trial153.stdout:1036: done (3 left) 2026-02-20T12:03:36.533 INFO:tasks.rados.rados.0.trial153.stdout:1037: done (2 left) 2026-02-20T12:03:36.533 INFO:tasks.rados.rados.0.trial153.stdout:1039: done (1 left) 2026-02-20T12:03:36.533 INFO:tasks.rados.rados.0.trial153.stdout:1040: done (0 left) 2026-02-20T12:03:36.533 INFO:tasks.rados.rados.0.trial153.stdout:1041: write (excl) oid 17 current snap is 126 2026-02-20T12:03:36.534 INFO:tasks.rados.rados.0.trial153.stdout:1041: seq_num 508 ranges {537956=705790,1644029=772160} 2026-02-20T12:03:36.537 INFO:tasks.rados.rados.0.trial153.stdout:1041: writing trial15342161-17 from 537956 to 1243746 tid 1 2026-02-20T12:03:36.542 INFO:tasks.rados.rados.0.trial153.stdout:1041: writing trial15342161-17 from 1644029 to 2416189 tid 2 2026-02-20T12:03:36.542 INFO:tasks.rados.rados.0.trial153.stdout:1042: snap_create 2026-02-20T12:03:36.582 INFO:tasks.rados.rados.0.trial153.stdout:1041: finishing write tid 1 to trial15342161-17 2026-02-20T12:03:36.599 INFO:tasks.rados.rados.0.trial153.stdout:1041: finishing write tid 2 to trial15342161-17 2026-02-20T12:03:36.605 INFO:tasks.rados.rados.0.trial153.stdout:1041: finishing write tid 3 to trial15342161-17 2026-02-20T12:03:36.607 INFO:tasks.rados.rados.0.trial153.stdout:1041: finishing write tid 4 to trial15342161-17 2026-02-20T12:03:36.607 INFO:tasks.rados.rados.0.trial153.stdout:1041: oid 17 updating version 0 to 227 2026-02-20T12:03:36.607 INFO:tasks.rados.rados.0.trial153.stdout:1041: oid 17 version 227 is already newer than 227 2026-02-20T12:03:36.607 INFO:tasks.rados.rados.0.trial153.stdout:1041: oid 17 version 227 is already newer than 225 2026-02-20T12:03:36.607 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 17 v 227 (ObjNum 508 snap 126 seq_num 508) dirty exists 2026-02-20T12:03:36.607 INFO:tasks.rados.rados.0.trial153.stdout:1041: left oid 17 (ObjNum 508 snap 126 seq_num 508) 2026-02-20T12:03:37.533 INFO:tasks.rados.rados.0.trial153.stdout:1041: done (1 left) 2026-02-20T12:03:37.533 INFO:tasks.rados.rados.0.trial153.stdout:1042: done (0 left) 2026-02-20T12:03:37.533 INFO:tasks.rados.rados.0.trial153.stdout:1043: delete oid 13 current snap is 127 2026-02-20T12:03:37.571 INFO:tasks.rados.rados.0.trial153.stdout:1043: done (0 left) 2026-02-20T12:03:37.571 INFO:tasks.rados.rados.0.trial153.stdout:1044: snap_create 2026-02-20T12:03:38.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~264s 2026-02-20T12:03:38.412 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:38.491 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:38.485+0000 7fba58acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:38.493 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:38.489+0000 7fba58acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:38.519 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:38.532 INFO:tasks.rados.rados.0.trial153.stdout:1044: done (0 left) 2026-02-20T12:03:38.532 INFO:tasks.rados.rados.0.trial153.stdout:1045: snap_remove snap 121 2026-02-20T12:03:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:38.536 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:39.535 INFO:tasks.rados.rados.0.trial153.stdout:1045: done (0 left) 2026-02-20T12:03:39.535 INFO:tasks.rados.rados.0.trial153.stdout:1046: delete oid 1 current snap is 128 2026-02-20T12:03:39.603 INFO:tasks.rados.rados.0.trial153.stdout:1046: done (0 left) 2026-02-20T12:03:39.603 INFO:tasks.rados.rados.0.trial153.stdout:1047: delete oid 38 current snap is 128 2026-02-20T12:03:39.629 INFO:tasks.rados.rados.0.trial153.stdout:1047: done (0 left) 2026-02-20T12:03:39.630 INFO:tasks.rados.rados.0.trial153.stdout:1048: write (excl) oid 33 current snap is 128 2026-02-20T12:03:39.630 INFO:tasks.rados.rados.0.trial153.stdout:1048: seq_num 512 ranges {605184=723139,1877957=727227} 2026-02-20T12:03:39.634 INFO:tasks.rados.rados.0.trial153.stdout:1048: writing trial15342161-33 from 605184 to 1328323 tid 1 2026-02-20T12:03:39.639 INFO:tasks.rados.rados.0.trial153.stdout:1048: writing trial15342161-33 from 1877957 to 2605184 tid 2 2026-02-20T12:03:39.639 INFO:tasks.rados.rados.0.trial153.stdout:1049: snap_create 2026-02-20T12:03:39.671 INFO:tasks.rados.rados.0.trial153.stdout:1048: finishing write tid 1 to trial15342161-33 2026-02-20T12:03:39.686 INFO:tasks.rados.rados.0.trial153.stdout:1048: finishing write tid 2 to trial15342161-33 2026-02-20T12:03:39.692 INFO:tasks.rados.rados.0.trial153.stdout:1048: finishing write tid 3 to trial15342161-33 2026-02-20T12:03:39.694 INFO:tasks.rados.rados.0.trial153.stdout:1048: finishing write tid 4 to trial15342161-33 2026-02-20T12:03:39.694 INFO:tasks.rados.rados.0.trial153.stdout:1048: oid 33 updating version 0 to 89 2026-02-20T12:03:39.694 INFO:tasks.rados.rados.0.trial153.stdout:1048: oid 33 updating version 89 to 91 2026-02-20T12:03:39.694 INFO:tasks.rados.rados.0.trial153.stdout:1048: oid 33 version 91 is already newer than 91 2026-02-20T12:03:39.694 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 33 v 91 (ObjNum 512 snap 128 seq_num 512) dirty exists 2026-02-20T12:03:39.695 INFO:tasks.rados.rados.0.trial153.stdout:1048: left oid 33 (ObjNum 512 snap 128 seq_num 512) 2026-02-20T12:03:40.538 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:40.615 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:40.610+0000 7fe926574640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:40.617 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:40.613+0000 7fe926574640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:40.625 INFO:tasks.rados.rados.0.trial153.stdout:1048: done (1 left) 2026-02-20T12:03:40.625 INFO:tasks.rados.rados.0.trial153.stdout:1049: done (0 left) 2026-02-20T12:03:40.625 INFO:tasks.rados.rados.0.trial153.stdout:1050: read oid 17 snap -1 2026-02-20T12:03:40.625 INFO:tasks.rados.rados.0.trial153.stdout:1050: expect (ObjNum 508 snap 126 seq_num 508) 2026-02-20T12:03:40.625 INFO:tasks.rados.rados.0.trial153.stdout:1051: read oid 9 snap -1 2026-02-20T12:03:40.625 INFO:tasks.rados.rados.0.trial153.stdout:1051: expect (ObjNum 498 snap 123 seq_num 498) 2026-02-20T12:03:40.626 INFO:tasks.rados.rados.0.trial153.stdout:1052: write (excl) oid 18 current snap is 129 2026-02-20T12:03:40.626 INFO:tasks.rados.rados.0.trial153.stdout:1052: seq_num 513 ranges {621991=798388,1936439=639509} 2026-02-20T12:03:40.631 INFO:tasks.rados.rados.0.trial153.stdout:1052: writing trial15342161-18 from 621991 to 1420379 tid 1 2026-02-20T12:03:40.636 INFO:tasks.rados.rados.0.trial153.stdout:1052: writing trial15342161-18 from 1936439 to 2575948 tid 2 2026-02-20T12:03:40.636 INFO:tasks.rados.rados.0.trial153.stdout:1053: snap_create 2026-02-20T12:03:40.652 INFO:tasks.rados.rados.0.trial153.stdout:1051: expect (ObjNum 498 snap 123 seq_num 498) 2026-02-20T12:03:40.661 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:40.671 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:40.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:40.712 INFO:tasks.rados.rados.0.trial153.stdout:1050: expect (ObjNum 508 snap 126 seq_num 508) 2026-02-20T12:03:40.712 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:40.750 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 890536 2026-02-20T12:03:40.778 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:40.814 INFO:tasks.rados.rados.0.trial153.stdout:1052: finishing write tid 1 to trial15342161-18 2026-02-20T12:03:40.814 INFO:tasks.rados.rados.0.trial153.stdout:1052: finishing write tid 2 to trial15342161-18 2026-02-20T12:03:40.814 INFO:tasks.rados.rados.0.trial153.stdout:1052: finishing write tid 3 to trial15342161-18 2026-02-20T12:03:40.814 INFO:tasks.rados.rados.0.trial153.stdout:1052: finishing write tid 4 to trial15342161-18 2026-02-20T12:03:40.814 INFO:tasks.rados.rados.0.trial153.stdout:1052: oid 18 updating version 0 to 82 2026-02-20T12:03:40.814 INFO:tasks.rados.rados.0.trial153.stdout:1052: oid 18 updating version 82 to 84 2026-02-20T12:03:40.815 INFO:tasks.rados.rados.0.trial153.stdout:1052: oid 18 version 84 is already newer than 84 2026-02-20T12:03:40.815 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 18 v 84 (ObjNum 513 snap 129 seq_num 513) dirty exists 2026-02-20T12:03:40.815 INFO:tasks.rados.rados.0.trial153.stdout:1052: left oid 18 (ObjNum 513 snap 129 seq_num 513) 2026-02-20T12:03:41.352 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:41.425 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:41.429 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:41.629 INFO:tasks.rados.rados.0.trial153.stdout:1050: done (3 left) 2026-02-20T12:03:41.629 INFO:tasks.rados.rados.0.trial153.stdout:1051: done (2 left) 2026-02-20T12:03:41.630 INFO:tasks.rados.rados.0.trial153.stdout:1052: done (1 left) 2026-02-20T12:03:41.630 INFO:tasks.rados.rados.0.trial153.stdout:1053: done (0 left) 2026-02-20T12:03:41.630 INFO:tasks.rados.rados.0.trial153.stdout:1054: write oid 39 current snap is 130 2026-02-20T12:03:41.630 INFO:tasks.rados.rados.0.trial153.stdout:1054: seq_num 514 ranges {638798=473637,1594921=783167} 2026-02-20T12:03:41.634 INFO:tasks.rados.rados.0.trial153.stdout:1054: writing trial15342161-39 from 638798 to 1112435 tid 1 2026-02-20T12:03:41.639 INFO:tasks.rados.rados.0.trial153.stdout:1054: writing trial15342161-39 from 1594921 to 2378088 tid 2 2026-02-20T12:03:41.639 INFO:tasks.rados.rados.0.trial153.stdout:1055: snap_remove snap 127 2026-02-20T12:03:42.633 INFO:tasks.rados.rados.0.trial153.stdout:1055: done (1 left) 2026-02-20T12:03:42.633 INFO:tasks.rados.rados.0.trial153.stdout:1056: delete oid 13 current snap is 130 2026-02-20T12:03:42.633 INFO:tasks.rados.rados.0.trial153.stdout:1054: finishing write tid 1 to trial15342161-39 2026-02-20T12:03:42.633 INFO:tasks.rados.rados.0.trial153.stdout:1054: finishing write tid 2 to trial15342161-39 2026-02-20T12:03:42.634 INFO:tasks.rados.rados.0.trial153.stdout:1054: finishing write tid 3 to trial15342161-39 2026-02-20T12:03:42.634 INFO:tasks.rados.rados.0.trial153.stdout:1054: finishing write tid 4 to trial15342161-39 2026-02-20T12:03:42.634 INFO:tasks.rados.rados.0.trial153.stdout:1054: oid 39 updating version 0 to 235 2026-02-20T12:03:42.634 INFO:tasks.rados.rados.0.trial153.stdout:1054: oid 39 version 235 is already newer than 235 2026-02-20T12:03:42.634 INFO:tasks.rados.rados.0.trial153.stdout:1054: oid 39 version 235 is already newer than 235 2026-02-20T12:03:42.634 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 39 v 235 (ObjNum 514 snap 130 seq_num 514) dirty exists 2026-02-20T12:03:42.634 INFO:tasks.rados.rados.0.trial153.stdout:1054: left oid 39 (ObjNum 514 snap 130 seq_num 514) 2026-02-20T12:03:42.648 INFO:tasks.rados.rados.0.trial153.stdout:1054: done (1 left) 2026-02-20T12:03:42.648 INFO:tasks.rados.rados.0.trial153.stdout:1056: done (0 left) 2026-02-20T12:03:42.648 INFO:tasks.rados.rados.0.trial153.stdout:1057: snap_remove snap 128 2026-02-20T12:03:42.679 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:42.754 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:42.749+0000 7f651010b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:42.757 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:42.752+0000 7f651010b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:42.783 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:42.800 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:43.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2026-02-20T12:03:43.638 INFO:tasks.rados.rados.0.trial153.stdout:1057: done (0 left) 2026-02-20T12:03:43.638 INFO:tasks.rados.rados.0.trial153.stdout:1058: snap_create 2026-02-20T12:03:44.639 INFO:tasks.rados.rados.0.trial153.stdout:1058: done (0 left) 2026-02-20T12:03:44.639 INFO:tasks.rados.rados.0.trial153.stdout:1059: read oid 43 snap -1 2026-02-20T12:03:44.639 INFO:tasks.rados.rados.0.trial153.stdout:1059: expect (ObjNum 477 snap 118 seq_num 477) 2026-02-20T12:03:44.639 INFO:tasks.rados.rados.0.trial153.stdout:1060: snap_remove snap 129 2026-02-20T12:03:44.801 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:44.878 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:44.873+0000 7fbb7970b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:44.881 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:44.877+0000 7fbb7970b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:44.909 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:44.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:45.643 INFO:tasks.rados.rados.0.trial153.stdout:1060: done (1 left) 2026-02-20T12:03:45.644 INFO:tasks.rados.rados.0.trial153.stdout:1061: read oid 45 snap -1 2026-02-20T12:03:45.644 INFO:tasks.rados.rados.0.trial153.stdout:1061: expect deleted 2026-02-20T12:03:45.644 INFO:tasks.rados.rados.0.trial153.stdout:1062: snap_remove snap 126 2026-02-20T12:03:46.429 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:46.502 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:46.505 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:46.536 INFO:tasks.rados.rados.0.trial153.stdout:1062: done (2 left) 2026-02-20T12:03:46.536 INFO:tasks.rados.rados.0.trial153.stdout:1063: read oid 7 snap -1 2026-02-20T12:03:46.536 INFO:tasks.rados.rados.0.trial153.stdout:1063: expect deleted 2026-02-20T12:03:46.537 INFO:tasks.rados.rados.0.trial153.stdout:1064: snap_remove snap 123 2026-02-20T12:03:46.927 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:47.006 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:47.000+0000 7f1d00bc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:47.008 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:47.004+0000 7f1d00bc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:47.035 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:47.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:47.537 INFO:tasks.rados.rados.0.trial153.stdout:1064: done (3 left) 2026-02-20T12:03:47.538 INFO:tasks.rados.rados.0.trial153.stdout:1065: read oid 18 snap -1 2026-02-20T12:03:47.538 INFO:tasks.rados.rados.0.trial153.stdout:1065: expect (ObjNum 513 snap 129 seq_num 513) 2026-02-20T12:03:47.538 INFO:tasks.rados.rados.0.trial153.stdout:1066: snap_remove snap 130 2026-02-20T12:03:48.540 INFO:tasks.rados.rados.0.trial153.stdout:1066: done (4 left) 2026-02-20T12:03:48.540 INFO:tasks.rados.rados.0.trial153.stdout:1067: read oid 42 snap 125 2026-02-20T12:03:48.540 INFO:tasks.rados.rados.0.trial153.stdout:1067: expect (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:48.540 INFO:tasks.rados.rados.0.trial153.stdout:1068: write oid 36 current snap is 131 2026-02-20T12:03:48.540 INFO:tasks.rados.rados.0.trial153.stdout:1068: seq_num 516 ranges {672412=624135,1995532=676880} 2026-02-20T12:03:48.545 INFO:tasks.rados.rados.0.trial153.stdout:1068: writing trial15342161-36 from 672412 to 1296547 tid 1 2026-02-20T12:03:48.552 INFO:tasks.rados.rados.0.trial153.stdout:1068: writing trial15342161-36 from 1995532 to 2672412 tid 2 2026-02-20T12:03:48.552 INFO:tasks.rados.rados.0.trial153.stdout:1069: read oid 22 snap -1 2026-02-20T12:03:48.552 INFO:tasks.rados.rados.0.trial153.stdout:1069: expect (ObjNum 473 snap 116 seq_num 473) 2026-02-20T12:03:48.552 INFO:tasks.rados.rados.0.trial153.stdout:1070: read oid 16 snap -1 2026-02-20T12:03:48.553 INFO:tasks.rados.rados.0.trial153.stdout:1070: expect deleted 2026-02-20T12:03:48.553 INFO:tasks.rados.rados.0.trial153.stdout:1071: snap_create 2026-02-20T12:03:48.553 INFO:tasks.rados.rados.0.trial153.stdout:1059: expect (ObjNum 477 snap 118 seq_num 477) 2026-02-20T12:03:48.553 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:48.632 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 367203 2026-02-20T12:03:48.669 INFO:tasks.rados.rados.0.trial153.stdout:1065: expect (ObjNum 513 snap 129 seq_num 513) 2026-02-20T12:03:48.669 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 36432 2026-02-20T12:03:48.819 INFO:tasks.rados.rados.0.trial153.stdout:1067: expect (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:48.834 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1672487 2026-02-20T12:03:48.843 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:48.874 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:48.905 INFO:tasks.rados.rados.0.trial153.stdout:1068: finishing write tid 1 to trial15342161-36 2026-02-20T12:03:48.906 INFO:tasks.rados.rados.0.trial153.stdout:1068: finishing write tid 2 to trial15342161-36 2026-02-20T12:03:48.906 INFO:tasks.rados.rados.0.trial153.stdout:1068: finishing write tid 3 to trial15342161-36 2026-02-20T12:03:48.906 INFO:tasks.rados.rados.0.trial153.stdout:1068: finishing write tid 4 to trial15342161-36 2026-02-20T12:03:48.906 INFO:tasks.rados.rados.0.trial153.stdout:1068: oid 36 updating version 0 to 94 2026-02-20T12:03:48.906 INFO:tasks.rados.rados.0.trial153.stdout:1068: oid 36 version 94 is already newer than 94 2026-02-20T12:03:48.906 INFO:tasks.rados.rados.0.trial153.stdout:1068: oid 36 version 94 is already newer than 94 2026-02-20T12:03:48.906 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 36 v 94 (ObjNum 516 snap 131 seq_num 516) dirty exists 2026-02-20T12:03:48.906 INFO:tasks.rados.rados.0.trial153.stdout:1068: left oid 36 (ObjNum 516 snap 131 seq_num 516) 2026-02-20T12:03:48.906 INFO:tasks.rados.rados.0.trial153.stdout:1069: expect (ObjNum 473 snap 116 seq_num 473) 2026-02-20T12:03:48.928 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2799825 2026-02-20T12:03:49.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~275s 2026-02-20T12:03:49.050 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:49.126 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:49.120+0000 7f2c67b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:49.129 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:49.124+0000 7f2c67b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:49.156 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:49.175 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:49.542 INFO:tasks.rados.rados.0.trial153.stdout:1059: done (8 left) 2026-02-20T12:03:49.542 INFO:tasks.rados.rados.0.trial153.stdout:1061: done (7 left) 2026-02-20T12:03:49.542 INFO:tasks.rados.rados.0.trial153.stdout:1063: done (6 left) 2026-02-20T12:03:49.542 INFO:tasks.rados.rados.0.trial153.stdout:1065: done (5 left) 2026-02-20T12:03:49.542 INFO:tasks.rados.rados.0.trial153.stdout:1067: done (4 left) 2026-02-20T12:03:49.542 INFO:tasks.rados.rados.0.trial153.stdout:1068: done (3 left) 2026-02-20T12:03:49.542 INFO:tasks.rados.rados.0.trial153.stdout:1069: done (2 left) 2026-02-20T12:03:49.542 INFO:tasks.rados.rados.0.trial153.stdout:1070: done (1 left) 2026-02-20T12:03:49.542 INFO:tasks.rados.rados.0.trial153.stdout:1071: done (0 left) 2026-02-20T12:03:49.542 INFO:tasks.rados.rados.0.trial153.stdout:1072: snap_remove snap 122 2026-02-20T12:03:50.546 INFO:tasks.rados.rados.0.trial153.stdout:1072: done (0 left) 2026-02-20T12:03:50.546 INFO:tasks.rados.rados.0.trial153.stdout:1073: snap_create 2026-02-20T12:03:51.176 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:51.254 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:51.248+0000 7fbf85f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:51.257 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:51.252+0000 7fbf85f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:51.283 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:51.299 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:51.506 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:51.548 INFO:tasks.rados.rados.0.trial153.stdout:1073: done (0 left) 2026-02-20T12:03:51.548 INFO:tasks.rados.rados.0.trial153.stdout:1074: snap_create 2026-02-20T12:03:51.577 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:51.580 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:51.581 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:52.550 INFO:tasks.rados.rados.0.trial153.stdout:1074: done (0 left) 2026-02-20T12:03:52.551 INFO:tasks.rados.rados.0.trial153.stdout:1075: delete oid 25 current snap is 134 2026-02-20T12:03:52.571 INFO:tasks.rados.rados.0.trial153.stdout:1075: done (0 left) 2026-02-20T12:03:52.571 INFO:tasks.rados.rados.0.trial153.stdout:1076: write oid 35 current snap is 134 2026-02-20T12:03:52.571 INFO:tasks.rados.rados.0.trial153.stdout:1076: seq_num 518 ranges {706026=490986,1828849=790505} 2026-02-20T12:03:52.575 INFO:tasks.rados.rados.0.trial153.stdout:1076: writing trial15342161-35 from 706026 to 1197012 tid 1 2026-02-20T12:03:52.581 INFO:tasks.rados.rados.0.trial153.stdout:1076: writing trial15342161-35 from 1828849 to 2619354 tid 2 2026-02-20T12:03:52.582 INFO:tasks.rados.rados.0.trial153.stdout:1077: read oid 12 snap -1 2026-02-20T12:03:52.582 INFO:tasks.rados.rados.0.trial153.stdout:1077: expect (ObjNum 358 snap 85 seq_num 358) 2026-02-20T12:03:52.582 INFO:tasks.rados.rados.0.trial153.stdout:1078: read oid 42 snap -1 2026-02-20T12:03:52.582 INFO:tasks.rados.rados.0.trial153.stdout:1078: expect (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:52.582 INFO:tasks.rados.rados.0.trial153.stdout:1079: write oid 38 current snap is 134 2026-02-20T12:03:52.582 INFO:tasks.rados.rados.0.trial153.stdout:1079: seq_num 519 ranges {722833=566235,1887331=650516} 2026-02-20T12:03:52.586 INFO:tasks.rados.rados.0.trial153.stdout:1079: writing trial15342161-38 from 722833 to 1289068 tid 1 2026-02-20T12:03:52.591 INFO:tasks.rados.rados.0.trial153.stdout:1079: writing trial15342161-38 from 1887331 to 2537847 tid 2 2026-02-20T12:03:52.591 INFO:tasks.rados.rados.0.trial153.stdout:1080: read oid 11 snap -1 2026-02-20T12:03:52.591 INFO:tasks.rados.rados.0.trial153.stdout:1080: expect (ObjNum 377 snap 89 seq_num 377) 2026-02-20T12:03:52.591 INFO:tasks.rados.rados.0.trial153.stdout:1081: snap_create 2026-02-20T12:03:52.605 INFO:tasks.rados.rados.0.trial153.stdout:1077: expect (ObjNum 358 snap 85 seq_num 358) 2026-02-20T12:03:52.605 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:52.623 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 190652 2026-02-20T12:03:52.648 INFO:tasks.rados.rados.0.trial153.stdout:1078: expect (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:52.671 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:52.734 INFO:tasks.rados.rados.0.trial153.stdout:1080: expect (ObjNum 377 snap 89 seq_num 377) 2026-02-20T12:03:52.734 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:52.755 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:52.795 INFO:tasks.rados.rados.0.trial153.stdout:1079: finishing write tid 1 to trial15342161-38 2026-02-20T12:03:52.795 INFO:tasks.rados.rados.0.trial153.stdout:1079: finishing write tid 2 to trial15342161-38 2026-02-20T12:03:52.795 INFO:tasks.rados.rados.0.trial153.stdout:1079: finishing write tid 3 to trial15342161-38 2026-02-20T12:03:52.795 INFO:tasks.rados.rados.0.trial153.stdout:1079: finishing write tid 4 to trial15342161-38 2026-02-20T12:03:52.795 INFO:tasks.rados.rados.0.trial153.stdout:1079: oid 38 updating version 0 to 153 2026-02-20T12:03:52.795 INFO:tasks.rados.rados.0.trial153.stdout:1079: oid 38 version 153 is already newer than 153 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:1079: oid 38 version 153 is already newer than 153 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 38 v 153 (ObjNum 519 snap 134 seq_num 519) dirty exists 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:1079: left oid 38 (ObjNum 519 snap 134 seq_num 519) 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:1076: finishing write tid 1 to trial15342161-35 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:1076: finishing write tid 2 to trial15342161-35 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:1076: finishing write tid 3 to trial15342161-35 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:1076: finishing write tid 4 to trial15342161-35 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:1076: oid 35 updating version 0 to 206 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:1076: oid 35 version 206 is already newer than 206 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:1076: oid 35 version 206 is already newer than 206 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 35 v 206 (ObjNum 518 snap 134 seq_num 518) dirty exists 2026-02-20T12:03:52.796 INFO:tasks.rados.rados.0.trial153.stdout:1076: left oid 35 (ObjNum 518 snap 134 seq_num 518) 2026-02-20T12:03:53.300 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:53.378 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:53.372+0000 7fd8e8851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:53.380 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:53.376+0000 7fd8e8851640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:53.409 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:53.426 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:53.553 INFO:tasks.rados.rados.0.trial153.stdout:1076: done (5 left) 2026-02-20T12:03:53.553 INFO:tasks.rados.rados.0.trial153.stdout:1077: done (4 left) 2026-02-20T12:03:53.553 INFO:tasks.rados.rados.0.trial153.stdout:1078: done (3 left) 2026-02-20T12:03:53.554 INFO:tasks.rados.rados.0.trial153.stdout:1079: done (2 left) 2026-02-20T12:03:53.554 INFO:tasks.rados.rados.0.trial153.stdout:1080: done (1 left) 2026-02-20T12:03:53.554 INFO:tasks.rados.rados.0.trial153.stdout:1081: done (0 left) 2026-02-20T12:03:53.554 INFO:tasks.rados.rados.0.trial153.stdout:1082: delete oid 50 current snap is 135 2026-02-20T12:03:53.608 INFO:tasks.rados.rados.0.trial153.stdout:1082: done (0 left) 2026-02-20T12:03:53.608 INFO:tasks.rados.rados.0.trial153.stdout:1083: delete oid 28 current snap is 135 2026-02-20T12:03:53.610 INFO:tasks.rados.rados.0.trial153.stdout:1083: done (0 left) 2026-02-20T12:03:53.610 INFO:tasks.rados.rados.0.trial153.stdout:1084: write (excl) oid 42 current snap is 135 2026-02-20T12:03:53.610 INFO:tasks.rados.rados.0.trial153.stdout:1084: seq_num 522 ranges {773254=791982,2346424=426830} 2026-02-20T12:03:53.617 INFO:tasks.rados.rados.0.trial153.stdout:1084: writing trial15342161-42 from 773254 to 1565236 tid 1 2026-02-20T12:03:53.621 INFO:tasks.rados.rados.0.trial153.stdout:1084: writing trial15342161-42 from 2346424 to 2773254 tid 2 2026-02-20T12:03:53.622 INFO:tasks.rados.rados.0.trial153.stdout:1085: snap_remove snap 131 2026-02-20T12:03:54.513 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~281s 2026-02-20T12:03:54.557 INFO:tasks.rados.rados.0.trial153.stdout:1085: done (1 left) 2026-02-20T12:03:54.557 INFO:tasks.rados.rados.0.trial153.stdout:1086: write oid 48 current snap is 135 2026-02-20T12:03:54.557 INFO:tasks.rados.rados.0.trial153.stdout:1086: seq_num 523 ranges {790061=467231,2004906=541501} 2026-02-20T12:03:54.562 INFO:tasks.rados.rados.0.trial153.stdout:1086: writing trial15342161-48 from 790061 to 1257292 tid 1 2026-02-20T12:03:54.567 INFO:tasks.rados.rados.0.trial153.stdout:1086: writing trial15342161-48 from 2004906 to 2546407 tid 2 2026-02-20T12:03:54.567 INFO:tasks.rados.rados.0.trial153.stdout:1087: read oid 20 snap -1 2026-02-20T12:03:54.567 INFO:tasks.rados.rados.0.trial153.stdout:1087: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:03:54.567 INFO:tasks.rados.rados.0.trial153.stdout:1088: write (excl) oid 24 current snap is 135 2026-02-20T12:03:54.567 INFO:tasks.rados.rados.0.trial153.stdout:1084: finishing write tid 1 to trial15342161-42 2026-02-20T12:03:54.567 INFO:tasks.rados.rados.0.trial153.stdout:1088: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2026-02-20T12:03:54.571 INFO:tasks.rados.rados.0.trial153.stdout:1088: writing trial15342161-24 from 406868 to 949348 tid 1 2026-02-20T12:03:54.575 INFO:tasks.rados.rados.0.trial153.stdout:1088: writing trial15342161-24 from 1547035 to 1948547 tid 2 2026-02-20T12:03:54.578 INFO:tasks.rados.rados.0.trial153.stdout:1088: writing trial15342161-24 from 2690354 to 2806868 tid 3 2026-02-20T12:03:54.578 INFO:tasks.rados.rados.0.trial153.stdout:1089: delete oid 1 current snap is 135 2026-02-20T12:03:54.578 INFO:tasks.rados.rados.0.trial153.stdout:1084: finishing write tid 2 to trial15342161-42 2026-02-20T12:03:54.578 INFO:tasks.rados.rados.0.trial153.stdout:1084: finishing write tid 3 to trial15342161-42 2026-02-20T12:03:54.578 INFO:tasks.rados.rados.0.trial153.stdout:1084: finishing write tid 4 to trial15342161-42 2026-02-20T12:03:54.578 INFO:tasks.rados.rados.0.trial153.stdout:1084: oid 42 updating version 0 to 89 2026-02-20T12:03:54.578 INFO:tasks.rados.rados.0.trial153.stdout:1084: oid 42 updating version 89 to 91 2026-02-20T12:03:54.579 INFO:tasks.rados.rados.0.trial153.stdout:1084: oid 42 version 91 is already newer than 91 2026-02-20T12:03:54.579 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 42 v 91 (ObjNum 522 snap 135 seq_num 522) dirty exists 2026-02-20T12:03:54.579 INFO:tasks.rados.rados.0.trial153.stdout:1084: left oid 42 (ObjNum 522 snap 135 seq_num 522) 2026-02-20T12:03:54.583 INFO:tasks.rados.rados.0.trial153.stdout:1084: done (4 left) 2026-02-20T12:03:54.583 INFO:tasks.rados.rados.0.trial153.stdout:1089: done (3 left) 2026-02-20T12:03:54.583 INFO:tasks.rados.rados.0.trial153.stdout:1090: write (excl) oid 34 current snap is 135 2026-02-20T12:03:54.583 INFO:tasks.rados.rados.0.trial153.stdout:1090: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2026-02-20T12:03:54.586 INFO:tasks.rados.rados.0.trial153.stdout:1090: writing trial15342161-34 from 440482 to 849813 tid 1 2026-02-20T12:03:54.590 INFO:tasks.rados.rados.0.trial153.stdout:1090: writing trial15342161-34 from 1380352 to 1785533 tid 2 2026-02-20T12:03:54.594 INFO:tasks.rados.rados.0.trial153.stdout:1090: writing trial15342161-34 from 2461553 to 2840482 tid 3 2026-02-20T12:03:54.594 INFO:tasks.rados.rados.0.trial153.stdout:1091: write (excl) oid 37 current snap is 135 2026-02-20T12:03:54.594 INFO:tasks.rados.rados.0.trial153.stdout:1091: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2026-02-20T12:03:54.597 INFO:tasks.rados.rados.0.trial153.stdout:1091: writing trial15342161-37 from 457289 to 941869 tid 1 2026-02-20T12:03:54.601 INFO:tasks.rados.rados.0.trial153.stdout:1091: writing trial15342161-37 from 1438834 to 1987673 tid 2 2026-02-20T12:03:54.605 INFO:tasks.rados.rados.0.trial153.stdout:1091: writing trial15342161-37 from 2772623 to 2857289 tid 3 2026-02-20T12:03:54.605 INFO:tasks.rados.rados.0.trial153.stdout:1092: write oid 46 current snap is 135 2026-02-20T12:03:54.605 INFO:tasks.rados.rados.0.trial153.stdout:1086: finishing write tid 1 to trial15342161-48 2026-02-20T12:03:54.605 INFO:tasks.rados.rados.0.trial153.stdout:1088: finishing write tid 1 to trial15342161-24 2026-02-20T12:03:54.605 INFO:tasks.rados.rados.0.trial153.stdout:1092: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2026-02-20T12:03:54.609 INFO:tasks.rados.rados.0.trial153.stdout:1092: writing trial15342161-46 from 474096 to 1033925 tid 1 2026-02-20T12:03:54.613 INFO:tasks.rados.rados.0.trial153.stdout:1092: writing trial15342161-46 from 1780963 to 2189813 tid 2 2026-02-20T12:03:54.616 INFO:tasks.rados.rados.0.trial153.stdout:1092: writing trial15342161-46 from 2800046 to 2874096 tid 3 2026-02-20T12:03:54.616 INFO:tasks.rados.rados.0.trial153.stdout:1093: write (excl) oid 1 current snap is 135 2026-02-20T12:03:54.616 INFO:tasks.rados.rados.0.trial153.stdout:1090: finishing write tid 1 to trial15342161-34 2026-02-20T12:03:54.616 INFO:tasks.rados.rados.0.trial153.stdout:1093: seq_num 529 ranges {490903=635078,1839445=552508} 2026-02-20T12:03:54.620 INFO:tasks.rados.rados.0.trial153.stdout:1093: writing trial15342161-1 from 490903 to 1125981 tid 1 2026-02-20T12:03:54.625 INFO:tasks.rados.rados.0.trial153.stdout:1093: writing trial15342161-1 from 1839445 to 2391953 tid 2 2026-02-20T12:03:54.625 INFO:tasks.rados.rados.0.trial153.stdout:1094: snap_create 2026-02-20T12:03:54.625 INFO:tasks.rados.rados.0.trial153.stdout:1087: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:03:54.625 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:03:54.788 INFO:tasks.rados.rados.0.trial153.stdout:1091: finishing write tid 1 to trial15342161-37 2026-02-20T12:03:54.788 INFO:tasks.rados.rados.0.trial153.stdout:1086: finishing write tid 2 to trial15342161-48 2026-02-20T12:03:54.788 INFO:tasks.rados.rados.0.trial153.stdout:1086: finishing write tid 3 to trial15342161-48 2026-02-20T12:03:54.788 INFO:tasks.rados.rados.0.trial153.stdout:1088: finishing write tid 2 to trial15342161-24 2026-02-20T12:03:54.788 INFO:tasks.rados.rados.0.trial153.stdout:1086: finishing write tid 4 to trial15342161-48 2026-02-20T12:03:54.788 INFO:tasks.rados.rados.0.trial153.stdout:1086: oid 48 updating version 0 to 108 2026-02-20T12:03:54.788 INFO:tasks.rados.rados.0.trial153.stdout:1086: oid 48 version 108 is already newer than 108 2026-02-20T12:03:54.788 INFO:tasks.rados.rados.0.trial153.stdout:1086: oid 48 version 108 is already newer than 108 2026-02-20T12:03:54.788 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 48 v 108 (ObjNum 523 snap 135 seq_num 523) dirty exists 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1086: left oid 48 (ObjNum 523 snap 135 seq_num 523) 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1088: finishing write tid 3 to trial15342161-24 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1090: finishing write tid 2 to trial15342161-34 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1092: finishing write tid 1 to trial15342161-46 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1091: finishing write tid 2 to trial15342161-37 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1088: finishing write tid 4 to trial15342161-24 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1088: finishing write tid 5 to trial15342161-24 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1088: oid 24 updating version 0 to 38 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1088: oid 24 version 38 is already newer than 38 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1088: oid 24 version 38 is already newer than 38 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:1088: oid 24 version 38 is already newer than 35 2026-02-20T12:03:54.789 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 24 v 38 (ObjNum 524 snap 135 seq_num 524) dirty exists 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1088: left oid 24 (ObjNum 524 snap 135 seq_num 524) 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1092: finishing write tid 2 to trial15342161-46 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1090: finishing write tid 3 to trial15342161-34 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1091: finishing write tid 3 to trial15342161-37 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1092: finishing write tid 3 to trial15342161-46 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1091: finishing write tid 4 to trial15342161-37 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1091: finishing write tid 5 to trial15342161-37 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1091: oid 37 updating version 0 to 103 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1091: oid 37 updating version 103 to 106 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1091: oid 37 version 106 is already newer than 106 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1091: oid 37 version 106 is already newer than 106 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 106 (ObjNum 527 snap 135 seq_num 527) dirty exists 2026-02-20T12:03:54.790 INFO:tasks.rados.rados.0.trial153.stdout:1091: left oid 37 (ObjNum 527 snap 135 seq_num 527) 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1090: finishing write tid 4 to trial15342161-34 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1093: finishing write tid 1 to trial15342161-1 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1092: finishing write tid 4 to trial15342161-46 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1090: finishing write tid 5 to trial15342161-34 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1090: oid 34 updating version 0 to 185 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1090: oid 34 version 185 is already newer than 182 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1090: oid 34 version 185 is already newer than 185 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1090: oid 34 version 185 is already newer than 185 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 185 (ObjNum 526 snap 135 seq_num 526) dirty exists 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1090: left oid 34 (ObjNum 526 snap 135 seq_num 526) 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1092: finishing write tid 5 to trial15342161-46 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1092: oid 46 updating version 0 to 54 2026-02-20T12:03:54.791 INFO:tasks.rados.rados.0.trial153.stdout:1092: oid 46 version 54 is already newer than 54 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:1092: oid 46 version 54 is already newer than 54 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:1092: oid 46 version 54 is already newer than 54 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 46 v 54 (ObjNum 528 snap 135 seq_num 528) dirty exists 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:1092: left oid 46 (ObjNum 528 snap 135 seq_num 528) 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:1093: finishing write tid 2 to trial15342161-1 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:1093: finishing write tid 3 to trial15342161-1 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:1093: finishing write tid 4 to trial15342161-1 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:1093: oid 1 updating version 0 to 313 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:1093: oid 1 version 313 is already newer than 313 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:1093: oid 1 version 313 is already newer than 311 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 1 v 313 (ObjNum 529 snap 135 seq_num 529) dirty exists 2026-02-20T12:03:54.792 INFO:tasks.rados.rados.0.trial153.stdout:1093: left oid 1 (ObjNum 529 snap 135 seq_num 529) 2026-02-20T12:03:55.427 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:55.506 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:55.501+0000 7fc357e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:55.509 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:55.504+0000 7fc357e5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:55.535 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:55.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:55.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:55.559 INFO:tasks.rados.rados.0.trial153.stdout:1086: done (7 left) 2026-02-20T12:03:55.559 INFO:tasks.rados.rados.0.trial153.stdout:1087: done (6 left) 2026-02-20T12:03:55.560 INFO:tasks.rados.rados.0.trial153.stdout:1088: done (5 left) 2026-02-20T12:03:55.560 INFO:tasks.rados.rados.0.trial153.stdout:1090: done (4 left) 2026-02-20T12:03:55.560 INFO:tasks.rados.rados.0.trial153.stdout:1091: done (3 left) 2026-02-20T12:03:55.560 INFO:tasks.rados.rados.0.trial153.stdout:1092: done (2 left) 2026-02-20T12:03:55.560 INFO:tasks.rados.rados.0.trial153.stdout:1093: done (1 left) 2026-02-20T12:03:55.560 INFO:tasks.rados.rados.0.trial153.stdout:1094: done (0 left) 2026-02-20T12:03:55.560 INFO:tasks.rados.rados.0.trial153.stdout:1095: write oid 43 current snap is 136 2026-02-20T12:03:55.560 INFO:tasks.rados.rados.0.trial153.stdout:1095: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2026-02-20T12:03:55.564 INFO:tasks.rados.rados.0.trial153.stdout:1095: writing trial15342161-43 from 507710 to 1218037 tid 1 2026-02-20T12:03:55.568 INFO:tasks.rados.rados.0.trial153.stdout:1095: writing trial15342161-43 from 1897927 to 2310446 tid 2 2026-02-20T12:03:55.570 INFO:tasks.rados.rados.0.trial153.stdout:1095: writing trial15342161-43 from 2854892 to 2907710 tid 3 2026-02-20T12:03:55.570 INFO:tasks.rados.rados.0.trial153.stdout:1096: snap_create 2026-02-20T12:03:55.605 INFO:tasks.rados.rados.0.trial153.stdout:1095: finishing write tid 1 to trial15342161-43 2026-02-20T12:03:55.612 INFO:tasks.rados.rados.0.trial153.stdout:1095: finishing write tid 2 to trial15342161-43 2026-02-20T12:03:55.617 INFO:tasks.rados.rados.0.trial153.stdout:1095: finishing write tid 3 to trial15342161-43 2026-02-20T12:03:55.619 INFO:tasks.rados.rados.0.trial153.stdout:1095: finishing write tid 4 to trial15342161-43 2026-02-20T12:03:55.621 INFO:tasks.rados.rados.0.trial153.stdout:1095: finishing write tid 5 to trial15342161-43 2026-02-20T12:03:55.621 INFO:tasks.rados.rados.0.trial153.stdout:1095: oid 43 updating version 0 to 92 2026-02-20T12:03:55.621 INFO:tasks.rados.rados.0.trial153.stdout:1095: oid 43 version 92 is already newer than 92 2026-02-20T12:03:55.621 INFO:tasks.rados.rados.0.trial153.stdout:1095: oid 43 version 92 is already newer than 92 2026-02-20T12:03:55.621 INFO:tasks.rados.rados.0.trial153.stdout:1095: oid 43 version 92 is already newer than 92 2026-02-20T12:03:55.621 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 43 v 92 (ObjNum 530 snap 136 seq_num 530) dirty exists 2026-02-20T12:03:55.621 INFO:tasks.rados.rados.0.trial153.stdout:1095: left oid 43 (ObjNum 530 snap 136 seq_num 530) 2026-02-20T12:03:56.537 INFO:tasks.rados.rados.0.trial153.stdout:1095: done (1 left) 2026-02-20T12:03:56.537 INFO:tasks.rados.rados.0.trial153.stdout:1096: done (0 left) 2026-02-20T12:03:56.537 INFO:tasks.rados.rados.0.trial153.stdout:1097: read oid 22 snap -1 2026-02-20T12:03:56.537 INFO:tasks.rados.rados.0.trial153.stdout:1097: expect (ObjNum 473 snap 116 seq_num 473) 2026-02-20T12:03:56.537 INFO:tasks.rados.rados.0.trial153.stdout:1098: read oid 14 snap 136 2026-02-20T12:03:56.538 INFO:tasks.rados.rados.0.trial153.stdout:1098: expect (ObjNum 466 snap 115 seq_num 466) 2026-02-20T12:03:56.538 INFO:tasks.rados.rados.0.trial153.stdout:1099: write (excl) oid 6 current snap is 137 2026-02-20T12:03:56.538 INFO:tasks.rados.rados.0.trial153.stdout:1099: seq_num 531 ranges {524517=785576,1956409=556177} 2026-02-20T12:03:56.541 INFO:tasks.rados.rados.0.trial153.stdout:1099: writing trial15342161-6 from 524517 to 1310093 tid 1 2026-02-20T12:03:56.546 INFO:tasks.rados.rados.0.trial153.stdout:1099: writing trial15342161-6 from 1956409 to 2512586 tid 2 2026-02-20T12:03:56.546 INFO:tasks.rados.rados.0.trial153.stdout:1100: delete oid 44 current snap is 137 2026-02-20T12:03:56.573 INFO:tasks.rados.rados.0.trial153.stdout:1097: expect (ObjNum 473 snap 116 seq_num 473) 2026-02-20T12:03:56.582 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:03:56.586 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1742082 2026-02-20T12:03:56.641 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2275253 2026-02-20T12:03:56.654 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:03:56.657 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:03:56.657 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:03:56.669 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 420055 2026-02-20T12:03:56.725 INFO:tasks.rados.rados.0.trial153.stdout:1097: done (3 left) 2026-02-20T12:03:56.725 INFO:tasks.rados.rados.0.trial153.stdout:1100: done (2 left) 2026-02-20T12:03:56.725 INFO:tasks.rados.rados.0.trial153.stdout:1101: read oid 31 snap -1 2026-02-20T12:03:56.725 INFO:tasks.rados.rados.0.trial153.stdout:1101: expect deleted 2026-02-20T12:03:56.725 INFO:tasks.rados.rados.0.trial153.stdout:1102: snap_remove snap 133 2026-02-20T12:03:57.551 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:03:57.563 INFO:tasks.rados.rados.0.trial153.stdout:1102: done (3 left) 2026-02-20T12:03:57.563 INFO:tasks.rados.rados.0.trial153.stdout:1103: read oid 42 snap 132 2026-02-20T12:03:57.564 INFO:tasks.rados.rados.0.trial153.stdout:1103: expect (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:57.564 INFO:tasks.rados.rados.0.trial153.stdout:1104: snap_create 2026-02-20T12:03:57.564 INFO:tasks.rados.rados.0.trial153.stdout:1098: expect (ObjNum 466 snap 115 seq_num 466) 2026-02-20T12:03:57.604 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 3038070 2026-02-20T12:03:57.632 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:57.626+0000 7fc1fe70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:57.634 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:57.630+0000 7fc1fe70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:57.634 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2456061 2026-02-20T12:03:57.663 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:57.675 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:57.676 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:03:57.730 INFO:tasks.rados.rados.0.trial153.stdout:1099: finishing write tid 1 to trial15342161-6 2026-02-20T12:03:57.730 INFO:tasks.rados.rados.0.trial153.stdout:1099: finishing write tid 2 to trial15342161-6 2026-02-20T12:03:57.730 INFO:tasks.rados.rados.0.trial153.stdout:1099: finishing write tid 3 to trial15342161-6 2026-02-20T12:03:57.731 INFO:tasks.rados.rados.0.trial153.stdout:1099: finishing write tid 4 to trial15342161-6 2026-02-20T12:03:57.731 INFO:tasks.rados.rados.0.trial153.stdout:1099: oid 6 updating version 0 to 334 2026-02-20T12:03:57.731 INFO:tasks.rados.rados.0.trial153.stdout:1099: oid 6 updating version 334 to 336 2026-02-20T12:03:57.731 INFO:tasks.rados.rados.0.trial153.stdout:1099: oid 6 version 336 is already newer than 336 2026-02-20T12:03:57.731 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 6 v 336 (ObjNum 531 snap 137 seq_num 531) dirty exists 2026-02-20T12:03:57.731 INFO:tasks.rados.rados.0.trial153.stdout:1099: left oid 6 (ObjNum 531 snap 137 seq_num 531) 2026-02-20T12:03:57.731 INFO:tasks.rados.rados.0.trial153.stdout:1103: expect (ObjNum 493 snap 122 seq_num 493) 2026-02-20T12:03:58.566 INFO:tasks.rados.rados.0.trial153.stdout:1098: done (4 left) 2026-02-20T12:03:58.566 INFO:tasks.rados.rados.0.trial153.stdout:1099: done (3 left) 2026-02-20T12:03:58.566 INFO:tasks.rados.rados.0.trial153.stdout:1101: done (2 left) 2026-02-20T12:03:58.566 INFO:tasks.rados.rados.0.trial153.stdout:1103: done (1 left) 2026-02-20T12:03:58.566 INFO:tasks.rados.rados.0.trial153.stdout:1104: done (0 left) 2026-02-20T12:03:58.566 INFO:tasks.rados.rados.0.trial153.stdout:1105: snap_remove snap 125 2026-02-20T12:03:59.572 INFO:tasks.rados.rados.0.trial153.stdout:1105: done (0 left) 2026-02-20T12:03:59.572 INFO:tasks.rados.rados.0.trial153.stdout:1106: read oid 36 snap -1 2026-02-20T12:03:59.573 INFO:tasks.rados.rados.0.trial153.stdout:1106: expect (ObjNum 516 snap 131 seq_num 516) 2026-02-20T12:03:59.573 INFO:tasks.rados.rados.0.trial153.stdout:1107: snap_remove snap 132 2026-02-20T12:03:59.677 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:03:59.753 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:59.748+0000 7f39e0f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:59.756 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:03:59.752+0000 7f39e0f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:03:59.784 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:03:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:03:59.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:04:00.017 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2026-02-20T12:04:00.576 INFO:tasks.rados.rados.0.trial153.stdout:1107: done (1 left) 2026-02-20T12:04:00.576 INFO:tasks.rados.rados.0.trial153.stdout:1108: write oid 37 current snap is 138 2026-02-20T12:04:00.576 INFO:tasks.rados.rados.0.trial153.stdout:1108: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2026-02-20T12:04:00.580 INFO:tasks.rados.rados.0.trial153.stdout:1108: writing trial15342161-37 from 558131 to 1210558 tid 1 2026-02-20T12:04:00.584 INFO:tasks.rados.rados.0.trial153.stdout:1108: writing trial15342161-37 from 1673373 to 2233219 tid 2 2026-02-20T12:04:00.587 INFO:tasks.rados.rados.0.trial153.stdout:1108: writing trial15342161-37 from 2820808 to 2958131 tid 3 2026-02-20T12:04:00.587 INFO:tasks.rados.rados.0.trial153.stdout:1109: read oid 17 snap -1 2026-02-20T12:04:00.587 INFO:tasks.rados.rados.0.trial153.stdout:1109: expect (ObjNum 508 snap 126 seq_num 508) 2026-02-20T12:04:00.587 INFO:tasks.rados.rados.0.trial153.stdout:1110: snap_create 2026-02-20T12:04:00.587 INFO:tasks.rados.rados.0.trial153.stdout:1106: expect (ObjNum 516 snap 131 seq_num 516) 2026-02-20T12:04:00.647 INFO:tasks.rados.rados.0.trial153.stdout:1109: expect (ObjNum 508 snap 126 seq_num 508) 2026-02-20T12:04:00.650 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 893707 2026-02-20T12:04:00.726 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2308692 2026-02-20T12:04:00.732 INFO:tasks.rados.rados.0.trial153.stdout:1108: finishing write tid 1 to trial15342161-37 2026-02-20T12:04:00.733 INFO:tasks.rados.rados.0.trial153.stdout:1108: finishing write tid 2 to trial15342161-37 2026-02-20T12:04:00.733 INFO:tasks.rados.rados.0.trial153.stdout:1108: finishing write tid 3 to trial15342161-37 2026-02-20T12:04:00.733 INFO:tasks.rados.rados.0.trial153.stdout:1108: finishing write tid 4 to trial15342161-37 2026-02-20T12:04:00.733 INFO:tasks.rados.rados.0.trial153.stdout:1108: finishing write tid 5 to trial15342161-37 2026-02-20T12:04:00.733 INFO:tasks.rados.rados.0.trial153.stdout:1108: oid 37 updating version 0 to 111 2026-02-20T12:04:00.733 INFO:tasks.rados.rados.0.trial153.stdout:1108: oid 37 version 111 is already newer than 111 2026-02-20T12:04:00.733 INFO:tasks.rados.rados.0.trial153.stdout:1108: oid 37 version 111 is already newer than 111 2026-02-20T12:04:00.733 INFO:tasks.rados.rados.0.trial153.stdout:1108: oid 37 version 111 is already newer than 111 2026-02-20T12:04:00.733 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 37 v 111 (ObjNum 533 snap 138 seq_num 533) dirty exists 2026-02-20T12:04:00.734 INFO:tasks.rados.rados.0.trial153.stdout:1108: left oid 37 (ObjNum 533 snap 138 seq_num 533) 2026-02-20T12:04:01.581 INFO:tasks.rados.rados.0.trial153.stdout:1106: done (3 left) 2026-02-20T12:04:01.581 INFO:tasks.rados.rados.0.trial153.stdout:1108: done (2 left) 2026-02-20T12:04:01.581 INFO:tasks.rados.rados.0.trial153.stdout:1109: done (1 left) 2026-02-20T12:04:01.581 INFO:tasks.rados.rados.0.trial153.stdout:1110: done (0 left) 2026-02-20T12:04:01.581 INFO:tasks.rados.rados.0.trial153.stdout:1111: read oid 43 snap 138 2026-02-20T12:04:01.581 INFO:tasks.rados.rados.0.trial153.stdout:1111: expect (ObjNum 530 snap 136 seq_num 530) 2026-02-20T12:04:01.581 INFO:tasks.rados.rados.0.trial153.stdout:1112: read oid 5 snap -1 2026-02-20T12:04:01.581 INFO:tasks.rados.rados.0.trial153.stdout:1112: expect (ObjNum 506 snap 125 seq_num 506) 2026-02-20T12:04:01.581 INFO:tasks.rados.rados.0.trial153.stdout:1113: write (excl) oid 32 current snap is 139 2026-02-20T12:04:01.581 INFO:tasks.rados.rados.0.trial153.stdout:1113: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2026-02-20T12:04:01.586 INFO:tasks.rados.rados.0.trial153.stdout:1113: writing trial15342161-32 from 574938 to 1302614 tid 1 2026-02-20T12:04:01.591 INFO:tasks.rados.rados.0.trial153.stdout:1113: writing trial15342161-32 from 1731855 to 2435359 tid 2 2026-02-20T12:04:01.593 INFO:tasks.rados.rados.0.trial153.stdout:1113: writing trial15342161-32 from 2848231 to 2974938 tid 3 2026-02-20T12:04:01.593 INFO:tasks.rados.rados.0.trial153.stdout:1114: read oid 1 snap 136 2026-02-20T12:04:01.593 INFO:tasks.rados.rados.0.trial153.stdout:1114: expect (ObjNum 529 snap 135 seq_num 529) 2026-02-20T12:04:01.593 INFO:tasks.rados.rados.0.trial153.stdout:1115: snap_create 2026-02-20T12:04:01.614 INFO:tasks.rados.rados.0.trial153.stdout:1114: expect (ObjNum 529 snap 135 seq_num 529) 2026-02-20T12:04:01.637 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2850978 2026-02-20T12:04:01.641 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:01.658 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:01.666 INFO:tasks.rados.rados.0.trial153.stdout:1112: expect (ObjNum 506 snap 125 seq_num 506) 2026-02-20T12:04:01.672 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1514984 2026-02-20T12:04:01.688 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:01.717 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:01.727 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:01.730 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:01.746 INFO:tasks.rados.rados.0.trial153.stdout:1111: expect (ObjNum 530 snap 136 seq_num 530) 2026-02-20T12:04:01.747 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:01.802 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:04:01.877 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:01.872+0000 7feadbb62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:01.880 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:01.875+0000 7feadbb62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:01.903 INFO:tasks.rados.rados.0.trial153.stdout:1113: finishing write tid 1 to trial15342161-32 2026-02-20T12:04:01.903 INFO:tasks.rados.rados.0.trial153.stdout:1113: finishing write tid 2 to trial15342161-32 2026-02-20T12:04:01.903 INFO:tasks.rados.rados.0.trial153.stdout:1113: finishing write tid 3 to trial15342161-32 2026-02-20T12:04:01.903 INFO:tasks.rados.rados.0.trial153.stdout:1113: finishing write tid 4 to trial15342161-32 2026-02-20T12:04:01.904 INFO:tasks.rados.rados.0.trial153.stdout:1113: finishing write tid 5 to trial15342161-32 2026-02-20T12:04:01.904 INFO:tasks.rados.rados.0.trial153.stdout:1113: oid 32 updating version 0 to 158 2026-02-20T12:04:01.904 INFO:tasks.rados.rados.0.trial153.stdout:1113: oid 32 version 158 is already newer than 155 2026-02-20T12:04:01.904 INFO:tasks.rados.rados.0.trial153.stdout:1113: oid 32 version 158 is already newer than 158 2026-02-20T12:04:01.904 INFO:tasks.rados.rados.0.trial153.stdout:1113: oid 32 version 158 is already newer than 158 2026-02-20T12:04:01.904 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 32 v 158 (ObjNum 534 snap 139 seq_num 534) dirty exists 2026-02-20T12:04:01.904 INFO:tasks.rados.rados.0.trial153.stdout:1113: left oid 32 (ObjNum 534 snap 139 seq_num 534) 2026-02-20T12:04:01.906 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:04:01.923 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:04:01.923 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:04:02.585 INFO:tasks.rados.rados.0.trial153.stdout:1111: done (4 left) 2026-02-20T12:04:02.586 INFO:tasks.rados.rados.0.trial153.stdout:1112: done (3 left) 2026-02-20T12:04:02.586 INFO:tasks.rados.rados.0.trial153.stdout:1113: done (2 left) 2026-02-20T12:04:02.586 INFO:tasks.rados.rados.0.trial153.stdout:1114: done (1 left) 2026-02-20T12:04:02.586 INFO:tasks.rados.rados.0.trial153.stdout:1115: done (0 left) 2026-02-20T12:04:02.586 INFO:tasks.rados.rados.0.trial153.stdout:1116: read oid 33 snap 137 2026-02-20T12:04:02.586 INFO:tasks.rados.rados.0.trial153.stdout:1116: expect (ObjNum 512 snap 128 seq_num 512) 2026-02-20T12:04:02.586 INFO:tasks.rados.rados.0.trial153.stdout:1117: write (excl) oid 12 current snap is 140 2026-02-20T12:04:02.586 INFO:tasks.rados.rados.0.trial153.stdout:1117: seq_num 535 ranges {591745=402925,1790337=563515} 2026-02-20T12:04:02.589 INFO:tasks.rados.rados.0.trial153.stdout:1117: writing trial15342161-12 from 591745 to 994670 tid 1 2026-02-20T12:04:02.594 INFO:tasks.rados.rados.0.trial153.stdout:1117: writing trial15342161-12 from 1790337 to 2353852 tid 2 2026-02-20T12:04:02.594 INFO:tasks.rados.rados.0.trial153.stdout:1118: write (excl) oid 46 current snap is 140 2026-02-20T12:04:02.594 INFO:tasks.rados.rados.0.trial153.stdout:1118: seq_num 536 ranges {608552=478174,1848819=707173} 2026-02-20T12:04:02.598 INFO:tasks.rados.rados.0.trial153.stdout:1118: writing trial15342161-46 from 608552 to 1086726 tid 1 2026-02-20T12:04:02.604 INFO:tasks.rados.rados.0.trial153.stdout:1118: writing trial15342161-46 from 1848819 to 2555992 tid 2 2026-02-20T12:04:02.604 INFO:tasks.rados.rados.0.trial153.stdout:1119: write (excl) oid 19 current snap is 140 2026-02-20T12:04:02.604 INFO:tasks.rados.rados.0.trial153.stdout:1119: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2026-02-20T12:04:02.608 INFO:tasks.rados.rados.0.trial153.stdout:1119: writing trial15342161-19 from 625359 to 1178782 tid 1 2026-02-20T12:04:02.613 INFO:tasks.rados.rados.0.trial153.stdout:1119: writing trial15342161-19 from 1790948 to 2358132 tid 2 2026-02-20T12:04:02.615 INFO:tasks.rados.rados.0.trial153.stdout:1119: writing trial15342161-19 from 2930500 to 3025359 tid 3 2026-02-20T12:04:02.616 INFO:tasks.rados.rados.0.trial153.stdout:1120: write oid 5 current snap is 140 2026-02-20T12:04:02.616 INFO:tasks.rados.rados.0.trial153.stdout:1120: seq_num 538 ranges {642166=628672,1849430=710842} 2026-02-20T12:04:02.620 INFO:tasks.rados.rados.0.trial153.stdout:1120: writing trial15342161-5 from 642166 to 1270838 tid 1 2026-02-20T12:04:02.625 INFO:tasks.rados.rados.0.trial153.stdout:1120: writing trial15342161-5 from 1849430 to 2560272 tid 2 2026-02-20T12:04:02.626 INFO:tasks.rados.rados.0.trial153.stdout:1121: write oid 28 current snap is 140 2026-02-20T12:04:02.626 INFO:tasks.rados.rados.0.trial153.stdout:1121: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2026-02-20T12:04:02.630 INFO:tasks.rados.rados.0.trial153.stdout:1121: writing trial15342161-28 from 658973 to 1362894 tid 1 2026-02-20T12:04:02.634 INFO:tasks.rados.rados.0.trial153.stdout:1121: writing trial15342161-28 from 1907912 to 2478765 tid 2 2026-02-20T12:04:02.637 INFO:tasks.rados.rados.0.trial153.stdout:1121: writing trial15342161-28 from 2985346 to 3058973 tid 3 2026-02-20T12:04:02.637 INFO:tasks.rados.rados.0.trial153.stdout:1122: delete oid 50 current snap is 140 2026-02-20T12:04:02.637 INFO:tasks.rados.rados.0.trial153.stdout:1116: expect (ObjNum 512 snap 128 seq_num 512) 2026-02-20T12:04:02.637 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:02.675 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 262721 2026-02-20T12:04:02.768 INFO:tasks.rados.rados.0.trial153.stdout:1116: done (6 left) 2026-02-20T12:04:02.768 INFO:tasks.rados.rados.0.trial153.stdout:1122: done (5 left) 2026-02-20T12:04:02.769 INFO:tasks.rados.rados.0.trial153.stdout:1123: snap_create 2026-02-20T12:04:02.769 INFO:tasks.rados.rados.0.trial153.stdout:1117: finishing write tid 1 to trial15342161-12 2026-02-20T12:04:02.769 INFO:tasks.rados.rados.0.trial153.stdout:1117: finishing write tid 2 to trial15342161-12 2026-02-20T12:04:02.769 INFO:tasks.rados.rados.0.trial153.stdout:1121: finishing write tid 1 to trial15342161-28 2026-02-20T12:04:02.769 INFO:tasks.rados.rados.0.trial153.stdout:1117: finishing write tid 3 to trial15342161-12 2026-02-20T12:04:02.769 INFO:tasks.rados.rados.0.trial153.stdout:1117: finishing write tid 4 to trial15342161-12 2026-02-20T12:04:02.769 INFO:tasks.rados.rados.0.trial153.stdout:1117: oid 12 updating version 0 to 56 2026-02-20T12:04:02.769 INFO:tasks.rados.rados.0.trial153.stdout:1117: oid 12 updating version 56 to 58 2026-02-20T12:04:02.769 INFO:tasks.rados.rados.0.trial153.stdout:1117: oid 12 version 58 is already newer than 58 2026-02-20T12:04:02.769 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 12 v 58 (ObjNum 535 snap 140 seq_num 535) dirty exists 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1117: left oid 12 (ObjNum 535 snap 140 seq_num 535) 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1121: finishing write tid 2 to trial15342161-28 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1120: finishing write tid 1 to trial15342161-5 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1118: finishing write tid 1 to trial15342161-46 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1121: finishing write tid 3 to trial15342161-28 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1120: finishing write tid 2 to trial15342161-5 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1121: finishing write tid 4 to trial15342161-28 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1120: finishing write tid 3 to trial15342161-5 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1118: finishing write tid 2 to trial15342161-46 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1121: finishing write tid 5 to trial15342161-28 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1121: oid 28 updating version 0 to 210 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1121: oid 28 version 210 is already newer than 210 2026-02-20T12:04:02.770 INFO:tasks.rados.rados.0.trial153.stdout:1121: oid 28 version 210 is already newer than 210 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:1121: oid 28 version 210 is already newer than 210 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 28 v 210 (ObjNum 539 snap 140 seq_num 539) dirty exists 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:1121: left oid 28 (ObjNum 539 snap 140 seq_num 539) 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:1120: finishing write tid 4 to trial15342161-5 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:1120: oid 5 updating version 0 to 115 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:1120: oid 5 version 115 is already newer than 115 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:1120: oid 5 version 115 is already newer than 115 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 5 v 115 (ObjNum 538 snap 140 seq_num 538) dirty exists 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:1120: left oid 5 (ObjNum 538 snap 140 seq_num 538) 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:1118: finishing write tid 3 to trial15342161-46 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:1118: finishing write tid 4 to trial15342161-46 2026-02-20T12:04:02.771 INFO:tasks.rados.rados.0.trial153.stdout:1118: oid 46 updating version 0 to 62 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1118: oid 46 version 62 is already newer than 62 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1118: oid 46 version 62 is already newer than 60 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 46 v 62 (ObjNum 536 snap 140 seq_num 536) dirty exists 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1118: left oid 46 (ObjNum 536 snap 140 seq_num 536) 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1119: finishing write tid 1 to trial15342161-19 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1119: finishing write tid 2 to trial15342161-19 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1119: finishing write tid 3 to trial15342161-19 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1119: finishing write tid 4 to trial15342161-19 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1119: finishing write tid 5 to trial15342161-19 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1119: oid 19 updating version 0 to 353 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1119: oid 19 version 353 is already newer than 350 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1119: oid 19 version 353 is already newer than 353 2026-02-20T12:04:02.772 INFO:tasks.rados.rados.0.trial153.stdout:1119: oid 19 version 353 is already newer than 353 2026-02-20T12:04:02.773 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 19 v 353 (ObjNum 537 snap 140 seq_num 537) dirty exists 2026-02-20T12:04:02.773 INFO:tasks.rados.rados.0.trial153.stdout:1119: left oid 19 (ObjNum 537 snap 140 seq_num 537) 2026-02-20T12:04:03.589 INFO:tasks.rados.rados.0.trial153.stdout:1117: done (5 left) 2026-02-20T12:04:03.589 INFO:tasks.rados.rados.0.trial153.stdout:1118: done (4 left) 2026-02-20T12:04:03.589 INFO:tasks.rados.rados.0.trial153.stdout:1119: done (3 left) 2026-02-20T12:04:03.589 INFO:tasks.rados.rados.0.trial153.stdout:1120: done (2 left) 2026-02-20T12:04:03.589 INFO:tasks.rados.rados.0.trial153.stdout:1121: done (1 left) 2026-02-20T12:04:03.589 INFO:tasks.rados.rados.0.trial153.stdout:1123: done (0 left) 2026-02-20T12:04:03.589 INFO:tasks.rados.rados.0.trial153.stdout:1124: write (excl) oid 45 current snap is 141 2026-02-20T12:04:03.589 INFO:tasks.rados.rados.0.trial153.stdout:1124: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2026-02-20T12:04:03.593 INFO:tasks.rados.rados.0.trial153.stdout:1124: writing trial15342161-45 from 692587 to 1263359 tid 1 2026-02-20T12:04:03.598 INFO:tasks.rados.rados.0.trial153.stdout:1124: writing trial15342161-45 from 2024876 to 2599398 tid 2 2026-02-20T12:04:03.600 INFO:tasks.rados.rados.0.trial153.stdout:1124: writing trial15342161-45 from 3040192 to 3092587 tid 3 2026-02-20T12:04:03.600 INFO:tasks.rados.rados.0.trial153.stdout:1125: snap_remove snap 139 2026-02-20T12:04:03.925 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:04:03.999 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:03.994+0000 7fdd01ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:04.002 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:03.998+0000 7fdd01ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:04.029 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:04:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:04:04.045 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:04:04.592 INFO:tasks.rados.rados.0.trial153.stdout:1125: done (1 left) 2026-02-20T12:04:04.592 INFO:tasks.rados.rados.0.trial153.stdout:1126: write oid 31 current snap is 141 2026-02-20T12:04:04.592 INFO:tasks.rados.rados.0.trial153.stdout:1124: finishing write tid 1 to trial15342161-45 2026-02-20T12:04:04.592 INFO:tasks.rados.rados.0.trial153.stdout:1126: seq_num 542 ranges {709394=646021,2083358=718180} 2026-02-20T12:04:04.597 INFO:tasks.rados.rados.0.trial153.stdout:1126: writing trial15342161-31 from 709394 to 1355415 tid 1 2026-02-20T12:04:04.603 INFO:tasks.rados.rados.0.trial153.stdout:1126: writing trial15342161-31 from 2083358 to 2801538 tid 2 2026-02-20T12:04:04.603 INFO:tasks.rados.rados.0.trial153.stdout:1127: read oid 40 snap -1 2026-02-20T12:04:04.603 INFO:tasks.rados.rados.0.trial153.stdout:1127: expect (ObjNum 491 snap 122 seq_num 491) 2026-02-20T12:04:04.603 INFO:tasks.rados.rados.0.trial153.stdout:1128: snap_remove snap 136 2026-02-20T12:04:04.603 INFO:tasks.rados.rados.0.trial153.stdout:1124: finishing write tid 2 to trial15342161-45 2026-02-20T12:04:05.521 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~292s 2026-02-20T12:04:05.593 INFO:tasks.rados.rados.0.trial153.stdout:1128: done (3 left) 2026-02-20T12:04:05.593 INFO:tasks.rados.rados.0.trial153.stdout:1129: snap_remove snap 135 2026-02-20T12:04:06.046 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:04:06.125 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:06.120+0000 7f5000c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:06.127 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:06.123+0000 7f5000c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:06.155 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:04:06.171 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:04:06.172 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:04:06.540 INFO:tasks.rados.rados.0.trial153.stdout:1129: done (3 left) 2026-02-20T12:04:06.540 INFO:tasks.rados.rados.0.trial153.stdout:1130: delete oid 30 current snap is 141 2026-02-20T12:04:06.540 INFO:tasks.rados.rados.0.trial153.stdout:1124: finishing write tid 3 to trial15342161-45 2026-02-20T12:04:06.540 INFO:tasks.rados.rados.0.trial153.stdout:1124: finishing write tid 4 to trial15342161-45 2026-02-20T12:04:06.540 INFO:tasks.rados.rados.0.trial153.stdout:1124: finishing write tid 5 to trial15342161-45 2026-02-20T12:04:06.541 INFO:tasks.rados.rados.0.trial153.stdout:1124: oid 45 updating version 0 to 96 2026-02-20T12:04:06.541 INFO:tasks.rados.rados.0.trial153.stdout:1124: oid 45 version 96 is already newer than 96 2026-02-20T12:04:06.541 INFO:tasks.rados.rados.0.trial153.stdout:1124: oid 45 version 96 is already newer than 93 2026-02-20T12:04:06.541 INFO:tasks.rados.rados.0.trial153.stdout:1124: oid 45 version 96 is already newer than 96 2026-02-20T12:04:06.541 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 45 v 96 (ObjNum 541 snap 141 seq_num 541) dirty exists 2026-02-20T12:04:06.541 INFO:tasks.rados.rados.0.trial153.stdout:1124: left oid 45 (ObjNum 541 snap 141 seq_num 541) 2026-02-20T12:04:06.541 INFO:tasks.rados.rados.0.trial153.stdout:1127: expect (ObjNum 491 snap 122 seq_num 491) 2026-02-20T12:04:06.594 INFO:tasks.rados.rados.0.trial153.stdout:1126: finishing write tid 1 to trial15342161-31 2026-02-20T12:04:06.594 INFO:tasks.rados.rados.0.trial153.stdout:1126: finishing write tid 2 to trial15342161-31 2026-02-20T12:04:06.594 INFO:tasks.rados.rados.0.trial153.stdout:1124: done (3 left) 2026-02-20T12:04:06.594 INFO:tasks.rados.rados.0.trial153.stdout:1127: done (2 left) 2026-02-20T12:04:06.594 INFO:tasks.rados.rados.0.trial153.stdout:1130: done (1 left) 2026-02-20T12:04:06.594 INFO:tasks.rados.rados.0.trial153.stdout:1131: snap_remove snap 134 2026-02-20T12:04:06.731 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:06.804 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:06.807 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:07.599 INFO:tasks.rados.rados.0.trial153.stdout:1131: done (1 left) 2026-02-20T12:04:07.599 INFO:tasks.rados.rados.0.trial153.stdout:1132: snap_create 2026-02-20T12:04:07.599 INFO:tasks.rados.rados.0.trial153.stdout:1126: finishing write tid 3 to trial15342161-31 2026-02-20T12:04:07.599 INFO:tasks.rados.rados.0.trial153.stdout:1126: finishing write tid 4 to trial15342161-31 2026-02-20T12:04:07.600 INFO:tasks.rados.rados.0.trial153.stdout:1126: oid 31 updating version 0 to 101 2026-02-20T12:04:07.600 INFO:tasks.rados.rados.0.trial153.stdout:1126: oid 31 version 101 is already newer than 101 2026-02-20T12:04:07.600 INFO:tasks.rados.rados.0.trial153.stdout:1126: oid 31 version 101 is already newer than 101 2026-02-20T12:04:07.600 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 31 v 101 (ObjNum 542 snap 141 seq_num 542) dirty exists 2026-02-20T12:04:07.600 INFO:tasks.rados.rados.0.trial153.stdout:1126: left oid 31 (ObjNum 542 snap 141 seq_num 542) 2026-02-20T12:04:08.172 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:04:08.250 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:08.244+0000 7f856c10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:08.252 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:08.248+0000 7f856c10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:08.279 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:04:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:04:08.295 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:04:08.600 INFO:tasks.rados.rados.0.trial153.stdout:1126: done (1 left) 2026-02-20T12:04:08.600 INFO:tasks.rados.rados.0.trial153.stdout:1132: done (0 left) 2026-02-20T12:04:08.600 INFO:tasks.rados.rados.0.trial153.stdout:1133: write (excl) oid 23 current snap is 142 2026-02-20T12:04:08.600 INFO:tasks.rados.rados.0.trial153.stdout:1133: seq_num 544 ranges {743008=796519,2200322=721849} 2026-02-20T12:04:08.605 INFO:tasks.rados.rados.0.trial153.stdout:1133: writing trial15342161-23 from 743008 to 1539527 tid 1 2026-02-20T12:04:08.611 INFO:tasks.rados.rados.0.trial153.stdout:1133: writing trial15342161-23 from 2200322 to 2922171 tid 2 2026-02-20T12:04:08.612 INFO:tasks.rados.rados.0.trial153.stdout:1134: read oid 33 snap 138 2026-02-20T12:04:08.612 INFO:tasks.rados.rados.0.trial153.stdout:1134: expect (ObjNum 512 snap 128 seq_num 512) 2026-02-20T12:04:08.612 INFO:tasks.rados.rados.0.trial153.stdout:1135: snap_create 2026-02-20T12:04:08.634 INFO:tasks.rados.rados.0.trial153.stdout:1133: finishing write tid 1 to trial15342161-23 2026-02-20T12:04:08.641 INFO:tasks.rados.rados.0.trial153.stdout:1134: expect (ObjNum 512 snap 128 seq_num 512) 2026-02-20T12:04:08.719 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:08.758 INFO:tasks.rados.rados.0.trial153.stdout:1133: finishing write tid 2 to trial15342161-23 2026-02-20T12:04:08.758 INFO:tasks.rados.rados.0.trial153.stdout:1133: finishing write tid 3 to trial15342161-23 2026-02-20T12:04:08.759 INFO:tasks.rados.rados.0.trial153.stdout:1133: finishing write tid 4 to trial15342161-23 2026-02-20T12:04:08.759 INFO:tasks.rados.rados.0.trial153.stdout:1133: oid 23 updating version 0 to 213 2026-02-20T12:04:08.759 INFO:tasks.rados.rados.0.trial153.stdout:1133: oid 23 version 213 is already newer than 213 2026-02-20T12:04:08.759 INFO:tasks.rados.rados.0.trial153.stdout:1133: oid 23 version 213 is already newer than 211 2026-02-20T12:04:08.759 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 23 v 213 (ObjNum 544 snap 142 seq_num 544) dirty exists 2026-02-20T12:04:08.759 INFO:tasks.rados.rados.0.trial153.stdout:1133: left oid 23 (ObjNum 544 snap 142 seq_num 544) 2026-02-20T12:04:09.604 INFO:tasks.rados.rados.0.trial153.stdout:1133: done (2 left) 2026-02-20T12:04:09.604 INFO:tasks.rados.rados.0.trial153.stdout:1134: done (1 left) 2026-02-20T12:04:09.604 INFO:tasks.rados.rados.0.trial153.stdout:1135: done (0 left) 2026-02-20T12:04:09.604 INFO:tasks.rados.rados.0.trial153.stdout:1136: write oid 49 current snap is 143 2026-02-20T12:04:09.604 INFO:tasks.rados.rados.0.trial153.stdout:1136: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2026-02-20T12:04:09.609 INFO:tasks.rados.rados.0.trial153.stdout:1136: writing trial15342161-49 from 759815 to 1231583 tid 1 2026-02-20T12:04:09.613 INFO:tasks.rados.rados.0.trial153.stdout:1136: writing trial15342161-49 from 1742451 to 2207958 tid 2 2026-02-20T12:04:09.617 INFO:tasks.rados.rados.0.trial153.stdout:1136: writing trial15342161-49 from 2917178 to 3159815 tid 3 2026-02-20T12:04:09.617 INFO:tasks.rados.rados.0.trial153.stdout:1137: write oid 48 current snap is 143 2026-02-20T12:04:09.617 INFO:tasks.rados.rados.0.trial153.stdout:1137: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2026-02-20T12:04:09.622 INFO:tasks.rados.rados.0.trial153.stdout:1137: writing trial15342161-48 from 776622 to 1323639 tid 1 2026-02-20T12:04:09.626 INFO:tasks.rados.rados.0.trial153.stdout:1137: writing trial15342161-48 from 1800933 to 2526451 tid 2 2026-02-20T12:04:09.629 INFO:tasks.rados.rados.0.trial153.stdout:1137: writing trial15342161-48 from 2944601 to 3176622 tid 3 2026-02-20T12:04:09.629 INFO:tasks.rados.rados.0.trial153.stdout:1138: write (excl) oid 22 current snap is 143 2026-02-20T12:04:09.629 INFO:tasks.rados.rados.0.trial153.stdout:1138: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2026-02-20T12:04:09.635 INFO:tasks.rados.rados.0.trial153.stdout:1138: writing trial15342161-22 from 793429 to 1415695 tid 1 2026-02-20T12:04:09.638 INFO:tasks.rados.rados.0.trial153.stdout:1138: writing trial15342161-22 from 1859415 to 2328591 tid 2 2026-02-20T12:04:09.641 INFO:tasks.rados.rados.0.trial153.stdout:1138: writing trial15342161-22 from 2972024 to 3193429 tid 3 2026-02-20T12:04:09.642 INFO:tasks.rados.rados.0.trial153.stdout:1139: snap_remove snap 141 2026-02-20T12:04:10.296 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:04:10.374 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:10.368+0000 7f5154635640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:10.377 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:10.372+0000 7f5154635640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:10.404 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:04:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:04:10.420 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:04:10.606 INFO:tasks.rados.rados.0.trial153.stdout:1139: done (3 left) 2026-02-20T12:04:10.606 INFO:tasks.rados.rados.0.trial153.stdout:1140: delete oid 32 current snap is 143 2026-02-20T12:04:10.606 INFO:tasks.rados.rados.0.trial153.stdout:1136: finishing write tid 1 to trial15342161-49 2026-02-20T12:04:10.606 INFO:tasks.rados.rados.0.trial153.stdout:1136: finishing write tid 2 to trial15342161-49 2026-02-20T12:04:10.606 INFO:tasks.rados.rados.0.trial153.stdout:1138: finishing write tid 1 to trial15342161-22 2026-02-20T12:04:10.606 INFO:tasks.rados.rados.0.trial153.stdout:1136: finishing write tid 3 to trial15342161-49 2026-02-20T12:04:10.606 INFO:tasks.rados.rados.0.trial153.stdout:1138: finishing write tid 2 to trial15342161-22 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1136: finishing write tid 4 to trial15342161-49 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1138: finishing write tid 3 to trial15342161-22 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1136: finishing write tid 5 to trial15342161-49 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1136: oid 49 updating version 0 to 259 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1136: oid 49 version 259 is already newer than 259 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1136: oid 49 version 259 is already newer than 259 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1136: oid 49 version 259 is already newer than 259 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 49 v 259 (ObjNum 545 snap 143 seq_num 545) dirty exists 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1136: left oid 49 (ObjNum 545 snap 143 seq_num 545) 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1138: finishing write tid 4 to trial15342161-22 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1138: finishing write tid 5 to trial15342161-22 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1138: oid 22 updating version 0 to 218 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1138: oid 22 version 218 is already newer than 218 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1138: oid 22 version 218 is already newer than 218 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1138: oid 22 version 218 is already newer than 215 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 22 v 218 (ObjNum 547 snap 143 seq_num 547) dirty exists 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1138: left oid 22 (ObjNum 547 snap 143 seq_num 547) 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1137: finishing write tid 1 to trial15342161-48 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1137: finishing write tid 2 to trial15342161-48 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1137: finishing write tid 3 to trial15342161-48 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1137: finishing write tid 4 to trial15342161-48 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1137: finishing write tid 5 to trial15342161-48 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1137: oid 48 updating version 0 to 113 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1137: oid 48 version 113 is already newer than 113 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1137: oid 48 version 113 is already newer than 113 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1137: oid 48 version 113 is already newer than 113 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 48 v 113 (ObjNum 546 snap 143 seq_num 546) dirty exists 2026-02-20T12:04:10.607 INFO:tasks.rados.rados.0.trial153.stdout:1137: left oid 48 (ObjNum 546 snap 143 seq_num 546) 2026-02-20T12:04:10.658 INFO:tasks.rados.rados.0.trial153.stdout:1136: done (3 left) 2026-02-20T12:04:10.659 INFO:tasks.rados.rados.0.trial153.stdout:1137: done (2 left) 2026-02-20T12:04:10.659 INFO:tasks.rados.rados.0.trial153.stdout:1138: done (1 left) 2026-02-20T12:04:10.659 INFO:tasks.rados.rados.0.trial153.stdout:1140: done (0 left) 2026-02-20T12:04:10.659 INFO:tasks.rados.rados.0.trial153.stdout:1141: read oid 19 snap -1 2026-02-20T12:04:10.659 INFO:tasks.rados.rados.0.trial153.stdout:1141: expect (ObjNum 537 snap 140 seq_num 537) 2026-02-20T12:04:10.659 INFO:tasks.rados.rados.0.trial153.stdout:1142: delete oid 16 current snap is 143 2026-02-20T12:04:10.661 INFO:tasks.rados.rados.0.trial153.stdout:1142: done (1 left) 2026-02-20T12:04:10.661 INFO:tasks.rados.rados.0.trial153.stdout:1143: snap_create 2026-02-20T12:04:10.691 INFO:tasks.rados.rados.0.trial153.stdout:1141: expect (ObjNum 537 snap 140 seq_num 537) 2026-02-20T12:04:10.691 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:10.831 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:11.024 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~297s 2026-02-20T12:04:11.613 INFO:tasks.rados.rados.0.trial153.stdout:1141: done (1 left) 2026-02-20T12:04:11.613 INFO:tasks.rados.rados.0.trial153.stdout:1143: done (0 left) 2026-02-20T12:04:11.613 INFO:tasks.rados.rados.0.trial153.stdout:1144: snap_remove snap 142 2026-02-20T12:04:11.808 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:11.882 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:11.885 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:11.886 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:12.421 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:04:12.500 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:12.494+0000 7fad9b03d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:12.502 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:12.498+0000 7fad9b03d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:12.529 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:04:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:04:12.544 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:04:12.617 INFO:tasks.rados.rados.0.trial153.stdout:1144: done (0 left) 2026-02-20T12:04:12.617 INFO:tasks.rados.rados.0.trial153.stdout:1145: snap_remove snap 138 2026-02-20T12:04:13.620 INFO:tasks.rados.rados.0.trial153.stdout:1145: done (0 left) 2026-02-20T12:04:13.620 INFO:tasks.rados.rados.0.trial153.stdout:1146: read oid 4 snap 140 2026-02-20T12:04:13.620 INFO:tasks.rados.rados.0.trial153.stdout:1146: expect deleted 2026-02-20T12:04:13.620 INFO:tasks.rados.rados.0.trial153.stdout:1147: write oid 11 current snap is 144 2026-02-20T12:04:13.620 INFO:tasks.rados.rados.0.trial153.stdout:1147: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2026-02-20T12:04:13.623 INFO:tasks.rados.rados.0.trial153.stdout:1147: writing trial15342161-11 from 443850 to 1008216 tid 1 2026-02-20T12:04:13.629 INFO:tasks.rados.rados.0.trial153.stdout:1147: writing trial15342161-11 from 1634861 to 2367717 tid 2 2026-02-20T12:04:13.633 INFO:tasks.rados.rados.0.trial153.stdout:1147: writing trial15342161-11 from 2770646 to 3243850 tid 3 2026-02-20T12:04:13.633 INFO:tasks.rados.rados.0.trial153.stdout:1148: read oid 6 snap -1 2026-02-20T12:04:13.633 INFO:tasks.rados.rados.0.trial153.stdout:1148: expect (ObjNum 531 snap 137 seq_num 531) 2026-02-20T12:04:13.633 INFO:tasks.rados.rados.0.trial153.stdout:1149: snap_remove snap 143 2026-02-20T12:04:14.545 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-20T12:04:14.620 INFO:tasks.rados.rados.0.trial153.stdout:1149: done (3 left) 2026-02-20T12:04:14.620 INFO:tasks.rados.rados.0.trial153.stdout:1150: write (excl) oid 30 current snap is 144 2026-02-20T12:04:14.620 INFO:tasks.rados.rados.0.trial153.stdout:1150: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2026-02-20T12:04:14.623 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:14.618+0000 7f1b41011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:14.624 INFO:tasks.rados.rados.0.trial153.stdout:1150: writing trial15342161-30 from 460657 to 1100272 tid 1 2026-02-20T12:04:14.626 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:14.622+0000 7f1b41011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:14.628 INFO:tasks.rados.rados.0.trial153.stdout:1150: writing trial15342161-30 from 1693343 to 2169857 tid 2 2026-02-20T12:04:14.633 INFO:tasks.rados.rados.0.trial153.stdout:1150: writing trial15342161-30 from 2681716 to 3260657 tid 3 2026-02-20T12:04:14.633 INFO:tasks.rados.rados.0.trial153.stdout:1151: write (excl) oid 34 current snap is 144 2026-02-20T12:04:14.633 INFO:tasks.rados.rados.0.trial153.stdout:1151: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2026-02-20T12:04:14.637 INFO:tasks.rados.rados.0.trial153.stdout:1151: writing trial15342161-34 from 477464 to 1192328 tid 1 2026-02-20T12:04:14.643 INFO:tasks.rados.rados.0.trial153.stdout:1151: writing trial15342161-34 from 1751825 to 2488350 tid 2 2026-02-20T12:04:14.646 INFO:tasks.rados.rados.0.trial153.stdout:1151: writing trial15342161-34 from 3225492 to 3277464 tid 3 2026-02-20T12:04:14.646 INFO:tasks.rados.rados.0.trial153.stdout:1152: write (excl) oid 15 current snap is 144 2026-02-20T12:04:14.646 INFO:tasks.rados.rados.0.trial153.stdout:1152: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2026-02-20T12:04:14.651 INFO:tasks.rados.rados.0.trial153.stdout:1152: writing trial15342161-15 from 494271 to 1284384 tid 1 2026-02-20T12:04:14.655 INFO:tasks.rados.rados.0.trial153.stdout:1152: writing trial15342161-15 from 1693954 to 2174137 tid 2 2026-02-20T12:04:14.659 INFO:tasks.rados.rados.0.trial153.stdout:1152: writing trial15342161-15 from 2620209 to 3132338 tid 3 2026-02-20T12:04:14.659 INFO:tasks.rados.rados.0.trial153.stdout:1153: read oid 33 snap -1 2026-02-20T12:04:14.660 INFO:tasks.rados.rados.0.trial153.stdout:1153: expect (ObjNum 512 snap 128 seq_num 512) 2026-02-20T12:04:14.660 INFO:tasks.rados.rados.0.trial153.stdout:1154: read oid 40 snap -1 2026-02-20T12:04:14.660 INFO:tasks.rados.rados.0.trial153.stdout:1154: expect (ObjNum 491 snap 122 seq_num 491) 2026-02-20T12:04:14.660 INFO:tasks.rados.rados.0.trial153.stdout:1155: write (excl) oid 13 current snap is 144 2026-02-20T12:04:14.660 INFO:tasks.rados.rados.0.trial153.stdout:1155: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2026-02-20T12:04:14.663 INFO:tasks.rados.rados.0.trial153.stdout:1155: writing trial15342161-13 from 511078 to 976440 tid 1 2026-02-20T12:04:14.669 INFO:tasks.rados.rados.0.trial153.stdout:1155: writing trial15342161-13 from 1752436 to 2492630 tid 2 2026-02-20T12:04:14.672 INFO:tasks.rados.rados.0.trial153.stdout:1155: writing trial15342161-13 from 3163985 to 3311078 tid 3 2026-02-20T12:04:14.673 INFO:tasks.rados.rados.0.trial153.stdout:1146: done (8 left) 2026-02-20T12:04:14.673 INFO:tasks.rados.rados.0.trial153.stdout:1156: write oid 29 current snap is 144 2026-02-20T12:04:14.673 INFO:tasks.rados.rados.0.trial153.stdout:1156: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2026-02-20T12:04:14.677 INFO:tasks.rados.rados.0.trial153.stdout:1156: writing trial15342161-29 from 527885 to 1184849 tid 1 2026-02-20T12:04:14.681 INFO:tasks.rados.rados.0.trial153.stdout:1156: writing trial15342161-29 from 1927271 to 2411123 tid 2 2026-02-20T12:04:14.683 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:04:14.685 INFO:tasks.rados.rados.0.trial153.stdout:1156: writing trial15342161-29 from 3191408 to 3327885 tid 3 2026-02-20T12:04:14.685 INFO:tasks.rados.rados.0.trial153.stdout:1157: write (excl) oid 22 current snap is 144 2026-02-20T12:04:14.685 INFO:tasks.rados.rados.0.trial153.stdout:1148: expect (ObjNum 531 snap 137 seq_num 531) 2026-02-20T12:04:14.686 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:04:14.702 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:04:14.724 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:14.791 INFO:tasks.rados.rados.0.trial153.stdout:1147: finishing write tid 1 to trial15342161-11 2026-02-20T12:04:14.791 INFO:tasks.rados.rados.0.trial153.stdout:1147: finishing write tid 2 to trial15342161-11 2026-02-20T12:04:14.791 INFO:tasks.rados.rados.0.trial153.stdout:1157: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2026-02-20T12:04:14.796 INFO:tasks.rados.rados.0.trial153.stdout:1157: writing trial15342161-22 from 544692 to 1276905 tid 1 2026-02-20T12:04:14.801 INFO:tasks.rados.rados.0.trial153.stdout:1157: writing trial15342161-22 from 1985753 to 2729616 tid 2 2026-02-20T12:04:14.805 INFO:tasks.rados.rados.0.trial153.stdout:1157: writing trial15342161-22 from 3335184 to 3344692 tid 3 2026-02-20T12:04:14.805 INFO:tasks.rados.rados.0.trial153.stdout:1148: done (9 left) 2026-02-20T12:04:14.805 INFO:tasks.rados.rados.0.trial153.stdout:1158: read oid 49 snap -1 2026-02-20T12:04:14.805 INFO:tasks.rados.rados.0.trial153.stdout:1158: expect (ObjNum 545 snap 143 seq_num 545) 2026-02-20T12:04:14.805 INFO:tasks.rados.rados.0.trial153.stdout:1159: read oid 37 snap -1 2026-02-20T12:04:14.805 INFO:tasks.rados.rados.0.trial153.stdout:1159: expect (ObjNum 533 snap 138 seq_num 533) 2026-02-20T12:04:14.805 INFO:tasks.rados.rados.0.trial153.stdout:1160: delete oid 36 current snap is 144 2026-02-20T12:04:14.805 INFO:tasks.rados.rados.0.trial153.stdout:1147: finishing write tid 3 to trial15342161-11 2026-02-20T12:04:14.806 INFO:tasks.rados.rados.0.trial153.stdout:1147: finishing write tid 4 to trial15342161-11 2026-02-20T12:04:14.806 INFO:tasks.rados.rados.0.trial153.stdout:1147: finishing write tid 5 to trial15342161-11 2026-02-20T12:04:14.806 INFO:tasks.rados.rados.0.trial153.stdout:1147: oid 11 updating version 0 to 122 2026-02-20T12:04:14.806 INFO:tasks.rados.rados.0.trial153.stdout:1147: oid 11 version 122 is already newer than 122 2026-02-20T12:04:14.806 INFO:tasks.rados.rados.0.trial153.stdout:1147: oid 11 version 122 is already newer than 122 2026-02-20T12:04:14.806 INFO:tasks.rados.rados.0.trial153.stdout:1147: oid 11 version 122 is already newer than 122 2026-02-20T12:04:14.806 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 11 v 122 (ObjNum 550 snap 144 seq_num 550) dirty exists 2026-02-20T12:04:14.806 INFO:tasks.rados.rados.0.trial153.stdout:1147: left oid 11 (ObjNum 550 snap 144 seq_num 550) 2026-02-20T12:04:14.806 INFO:tasks.rados.rados.0.trial153.stdout:1150: finishing write tid 1 to trial15342161-30 2026-02-20T12:04:14.806 INFO:tasks.rados.rados.0.trial153.stdout:1154: expect (ObjNum 491 snap 122 seq_num 491) 2026-02-20T12:04:14.820 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:14.856 INFO:tasks.rados.rados.0.trial153.stdout:1150: finishing write tid 2 to trial15342161-30 2026-02-20T12:04:14.856 INFO:tasks.rados.rados.0.trial153.stdout:1155: finishing write tid 1 to trial15342161-13 2026-02-20T12:04:14.856 INFO:tasks.rados.rados.0.trial153.stdout:1153: expect (ObjNum 512 snap 128 seq_num 512) 2026-02-20T12:04:14.865 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1389771 2026-02-20T12:04:14.890 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:14.950 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 1522890 2026-02-20T12:04:14.967 INFO:tasks.rados.rados.0.trial153.stdout:1150: finishing write tid 3 to trial15342161-30 2026-02-20T12:04:14.967 INFO:tasks.rados.rados.0.trial153.stdout:1147: done (11 left) 2026-02-20T12:04:14.968 INFO:tasks.rados.rados.0.trial153.stdout:1153: done (10 left) 2026-02-20T12:04:14.968 INFO:tasks.rados.rados.0.trial153.stdout:1154: done (9 left) 2026-02-20T12:04:14.968 INFO:tasks.rados.rados.0.trial153.stdout:1160: done (8 left) 2026-02-20T12:04:14.968 INFO:tasks.rados.rados.0.trial153.stdout:1161: write (excl) oid 32 current snap is 144 2026-02-20T12:04:14.968 INFO:tasks.rados.rados.0.trial153.stdout:1155: finishing write tid 2 to trial15342161-13 2026-02-20T12:04:14.968 INFO:tasks.rados.rados.0.trial153.stdout:1161: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2026-02-20T12:04:14.971 INFO:tasks.rados.rados.0.trial153.stdout:1161: writing trial15342161-32 from 578306 to 1061017 tid 1 2026-02-20T12:04:14.975 INFO:tasks.rados.rados.0.trial153.stdout:1161: writing trial15342161-32 from 1586364 to 2217543 tid 2 2026-02-20T12:04:14.980 INFO:tasks.rados.rados.0.trial153.stdout:1161: writing trial15342161-32 from 2757324 to 3242166 tid 3 2026-02-20T12:04:14.980 INFO:tasks.rados.rados.0.trial153.stdout:1155: finishing write tid 3 to trial15342161-13 2026-02-20T12:04:14.981 INFO:tasks.rados.rados.0.trial153.stdout:1155: finishing write tid 4 to trial15342161-13 2026-02-20T12:04:14.981 INFO:tasks.rados.rados.0.trial153.stdout:1150: finishing write tid 4 to trial15342161-30 2026-02-20T12:04:14.981 INFO:tasks.rados.rados.0.trial153.stdout:1162: read oid 20 snap 137 2026-02-20T12:04:14.981 INFO:tasks.rados.rados.0.trial153.stdout:1162: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:04:14.981 INFO:tasks.rados.rados.0.trial153.stdout:1163: snap_create 2026-02-20T12:04:14.981 INFO:tasks.rados.rados.0.trial153.stdout:1155: finishing write tid 5 to trial15342161-13 2026-02-20T12:04:14.981 INFO:tasks.rados.rados.0.trial153.stdout:1155: oid 13 updating version 0 to 321 2026-02-20T12:04:14.981 INFO:tasks.rados.rados.0.trial153.stdout:1155: oid 13 version 321 is already newer than 318 2026-02-20T12:04:14.981 INFO:tasks.rados.rados.0.trial153.stdout:1155: oid 13 version 321 is already newer than 321 2026-02-20T12:04:14.981 INFO:tasks.rados.rados.0.trial153.stdout:1155: oid 13 version 321 is already newer than 321 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 13 v 321 (ObjNum 554 snap 144 seq_num 554) dirty exists 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:1155: left oid 13 (ObjNum 554 snap 144 seq_num 554) 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:1150: finishing write tid 5 to trial15342161-30 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:1150: oid 30 updating version 0 to 193 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:1150: oid 30 version 193 is already newer than 190 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:1150: oid 30 version 193 is already newer than 193 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:1150: oid 30 version 193 is already newer than 193 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 30 v 193 (ObjNum 551 snap 144 seq_num 551) dirty exists 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:1150: left oid 30 (ObjNum 551 snap 144 seq_num 551) 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:1156: finishing write tid 1 to trial15342161-29 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:1156: finishing write tid 2 to trial15342161-29 2026-02-20T12:04:14.982 INFO:tasks.rados.rados.0.trial153.stdout:1156: finishing write tid 3 to trial15342161-29 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:1156: finishing write tid 4 to trial15342161-29 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:1156: finishing write tid 5 to trial15342161-29 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:1156: oid 29 updating version 0 to 325 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:1156: oid 29 version 325 is already newer than 325 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:1156: oid 29 version 325 is already newer than 325 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:1156: oid 29 version 325 is already newer than 325 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 29 v 325 (ObjNum 555 snap 144 seq_num 555) dirty exists 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:1156: left oid 29 (ObjNum 555 snap 144 seq_num 555) 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:1151: finishing write tid 1 to trial15342161-34 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:1151: finishing write tid 2 to trial15342161-34 2026-02-20T12:04:14.983 INFO:tasks.rados.rados.0.trial153.stdout:1151: finishing write tid 3 to trial15342161-34 2026-02-20T12:04:14.984 INFO:tasks.rados.rados.0.trial153.stdout:1151: finishing write tid 4 to trial15342161-34 2026-02-20T12:04:14.984 INFO:tasks.rados.rados.0.trial153.stdout:1151: finishing write tid 5 to trial15342161-34 2026-02-20T12:04:14.984 INFO:tasks.rados.rados.0.trial153.stdout:1151: oid 34 updating version 0 to 195 2026-02-20T12:04:14.984 INFO:tasks.rados.rados.0.trial153.stdout:1151: oid 34 updating version 195 to 198 2026-02-20T12:04:14.984 INFO:tasks.rados.rados.0.trial153.stdout:1151: oid 34 version 198 is already newer than 198 2026-02-20T12:04:14.984 INFO:tasks.rados.rados.0.trial153.stdout:1151: oid 34 version 198 is already newer than 198 2026-02-20T12:04:14.984 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 198 (ObjNum 552 snap 144 seq_num 552) dirty exists 2026-02-20T12:04:14.984 INFO:tasks.rados.rados.0.trial153.stdout:1151: left oid 34 (ObjNum 552 snap 144 seq_num 552) 2026-02-20T12:04:14.984 INFO:tasks.rados.rados.0.trial153.stdout:1159: expect (ObjNum 533 snap 138 seq_num 533) 2026-02-20T12:04:14.994 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 2608121 2026-02-20T12:04:15.008 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:15.070 INFO:tasks.rados.rados.0.trial153.stdout:1157: finishing write tid 1 to trial15342161-22 2026-02-20T12:04:15.070 INFO:tasks.rados.rados.0.trial153.stdout:1158: expect (ObjNum 545 snap 143 seq_num 545) 2026-02-20T12:04:15.097 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 511375 2026-02-20T12:04:15.121 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:15.149 INFO:tasks.rados.rados.0.trial153.stdout:1152: finishing write tid 1 to trial15342161-15 2026-02-20T12:04:15.149 INFO:tasks.rados.rados.0.trial153.stdout:1157: finishing write tid 2 to trial15342161-22 2026-02-20T12:04:15.149 INFO:tasks.rados.rados.0.trial153.stdout:1152: finishing write tid 2 to trial15342161-15 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1152: finishing write tid 3 to trial15342161-15 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1152: finishing write tid 4 to trial15342161-15 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1152: finishing write tid 5 to trial15342161-15 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1152: oid 15 updating version 0 to 369 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1152: oid 15 updating version 369 to 372 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1152: oid 15 version 372 is already newer than 372 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1152: oid 15 version 372 is already newer than 372 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 15 v 372 (ObjNum 553 snap 144 seq_num 553) dirty exists 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1152: left oid 15 (ObjNum 553 snap 144 seq_num 553) 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1157: finishing write tid 3 to trial15342161-22 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1157: finishing write tid 4 to trial15342161-22 2026-02-20T12:04:15.150 INFO:tasks.rados.rados.0.trial153.stdout:1157: finishing write tid 5 to trial15342161-22 2026-02-20T12:04:15.151 INFO:tasks.rados.rados.0.trial153.stdout:1157: oid 22 updating version 0 to 222 2026-02-20T12:04:15.151 INFO:tasks.rados.rados.0.trial153.stdout:1157: oid 22 version 222 is already newer than 222 2026-02-20T12:04:15.151 INFO:tasks.rados.rados.0.trial153.stdout:1157: oid 22 version 222 is already newer than 222 2026-02-20T12:04:15.151 INFO:tasks.rados.rados.0.trial153.stdout:1157: oid 22 version 222 is already newer than 219 2026-02-20T12:04:15.151 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 22 v 222 (ObjNum 556 snap 144 seq_num 556) dirty exists 2026-02-20T12:04:15.151 INFO:tasks.rados.rados.0.trial153.stdout:1157: left oid 22 (ObjNum 556 snap 144 seq_num 556) 2026-02-20T12:04:15.151 INFO:tasks.rados.rados.0.trial153.stdout:1161: finishing write tid 1 to trial15342161-32 2026-02-20T12:04:15.151 INFO:tasks.rados.rados.0.trial153.stdout:1162: expect (ObjNum 429 snap 106 seq_num 429) 2026-02-20T12:04:15.202 INFO:tasks.rados.rados.0.trial153.stdout:seeking to 0 2026-02-20T12:04:15.299 INFO:tasks.rados.rados.0.trial153.stdout:1161: finishing write tid 2 to trial15342161-32 2026-02-20T12:04:15.299 INFO:tasks.rados.rados.0.trial153.stdout:1161: finishing write tid 3 to trial15342161-32 2026-02-20T12:04:15.299 INFO:tasks.rados.rados.0.trial153.stdout:1161: finishing write tid 4 to trial15342161-32 2026-02-20T12:04:15.299 INFO:tasks.rados.rados.0.trial153.stdout:1161: finishing write tid 5 to trial15342161-32 2026-02-20T12:04:15.299 INFO:tasks.rados.rados.0.trial153.stdout:1161: oid 32 updating version 0 to 164 2026-02-20T12:04:15.299 INFO:tasks.rados.rados.0.trial153.stdout:1161: oid 32 version 164 is already newer than 164 2026-02-20T12:04:15.299 INFO:tasks.rados.rados.0.trial153.stdout:1161: oid 32 version 164 is already newer than 164 2026-02-20T12:04:15.299 INFO:tasks.rados.rados.0.trial153.stdout:1161: oid 32 version 164 is already newer than 161 2026-02-20T12:04:15.300 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 32 v 164 (ObjNum 558 snap 144 seq_num 558) dirty exists 2026-02-20T12:04:15.300 INFO:tasks.rados.rados.0.trial153.stdout:1161: left oid 32 (ObjNum 558 snap 144 seq_num 558) 2026-02-20T12:04:15.625 INFO:tasks.rados.rados.0.trial153.stdout:1150: done (10 left) 2026-02-20T12:04:15.625 INFO:tasks.rados.rados.0.trial153.stdout:1151: done (9 left) 2026-02-20T12:04:15.626 INFO:tasks.rados.rados.0.trial153.stdout:1152: done (8 left) 2026-02-20T12:04:15.626 INFO:tasks.rados.rados.0.trial153.stdout:1155: done (7 left) 2026-02-20T12:04:15.626 INFO:tasks.rados.rados.0.trial153.stdout:1156: done (6 left) 2026-02-20T12:04:15.626 INFO:tasks.rados.rados.0.trial153.stdout:1157: done (5 left) 2026-02-20T12:04:15.626 INFO:tasks.rados.rados.0.trial153.stdout:1158: done (4 left) 2026-02-20T12:04:15.626 INFO:tasks.rados.rados.0.trial153.stdout:1159: done (3 left) 2026-02-20T12:04:15.626 INFO:tasks.rados.rados.0.trial153.stdout:1161: done (2 left) 2026-02-20T12:04:15.626 INFO:tasks.rados.rados.0.trial153.stdout:1162: done (1 left) 2026-02-20T12:04:15.626 INFO:tasks.rados.rados.0.trial153.stdout:1163: done (0 left) 2026-02-20T12:04:15.626 INFO:tasks.rados.rados.0.trial153.stdout:1164: write oid 34 current snap is 145 2026-02-20T12:04:15.627 INFO:tasks.rados.rados.0.trial153.stdout:1164: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2026-02-20T12:04:15.630 INFO:tasks.rados.rados.0.trial153.stdout:1164: writing trial15342161-34 from 595113 to 1153073 tid 1 2026-02-20T12:04:15.634 INFO:tasks.rados.rados.0.trial153.stdout:1164: writing trial15342161-34 from 1644846 to 2136036 tid 2 2026-02-20T12:04:15.638 INFO:tasks.rados.rados.0.trial153.stdout:1164: writing trial15342161-34 from 2784747 to 3395113 tid 3 2026-02-20T12:04:15.639 INFO:tasks.rados.rados.0.trial153.stdout:1165: write (excl) oid 29 current snap is 145 2026-02-20T12:04:15.639 INFO:tasks.rados.rados.0.trial153.stdout:1165: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2026-02-20T12:04:15.643 INFO:tasks.rados.rados.0.trial153.stdout:1165: writing trial15342161-29 from 611920 to 1245129 tid 1 2026-02-20T12:04:15.647 INFO:tasks.rados.rados.0.trial153.stdout:1165: writing trial15342161-29 from 1703328 to 2338176 tid 2 2026-02-20T12:04:15.651 INFO:tasks.rados.rados.0.trial153.stdout:1165: writing trial15342161-29 from 2812170 to 3319556 tid 3 2026-02-20T12:04:15.651 INFO:tasks.rados.rados.0.trial153.stdout:1166: snap_create 2026-02-20T12:04:15.677 INFO:tasks.rados.rados.0.trial153.stdout:1165: finishing write tid 1 to trial15342161-29 2026-02-20T12:04:15.689 INFO:tasks.rados.rados.0.trial153.stdout:1164: finishing write tid 1 to trial15342161-34 2026-02-20T12:04:15.694 INFO:tasks.rados.rados.0.trial153.stdout:1165: finishing write tid 2 to trial15342161-29 2026-02-20T12:04:15.698 INFO:tasks.rados.rados.0.trial153.stdout:1164: finishing write tid 2 to trial15342161-34 2026-02-20T12:04:15.700 INFO:tasks.rados.rados.0.trial153.stdout:1165: finishing write tid 3 to trial15342161-29 2026-02-20T12:04:15.709 INFO:tasks.rados.rados.0.trial153.stdout:1165: finishing write tid 4 to trial15342161-29 2026-02-20T12:04:15.711 INFO:tasks.rados.rados.0.trial153.stdout:1165: finishing write tid 5 to trial15342161-29 2026-02-20T12:04:15.711 INFO:tasks.rados.rados.0.trial153.stdout:1165: oid 29 updating version 0 to 327 2026-02-20T12:04:15.712 INFO:tasks.rados.rados.0.trial153.stdout:1165: oid 29 updating version 327 to 330 2026-02-20T12:04:15.712 INFO:tasks.rados.rados.0.trial153.stdout:1165: oid 29 version 330 is already newer than 330 2026-02-20T12:04:15.712 INFO:tasks.rados.rados.0.trial153.stdout:1165: oid 29 version 330 is already newer than 330 2026-02-20T12:04:15.712 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 29 v 330 (ObjNum 560 snap 145 seq_num 560) dirty exists 2026-02-20T12:04:15.712 INFO:tasks.rados.rados.0.trial153.stdout:1165: left oid 29 (ObjNum 560 snap 145 seq_num 560) 2026-02-20T12:04:15.714 INFO:tasks.rados.rados.0.trial153.stdout:1164: finishing write tid 3 to trial15342161-34 2026-02-20T12:04:15.718 INFO:tasks.rados.rados.0.trial153.stdout:1164: finishing write tid 4 to trial15342161-34 2026-02-20T12:04:15.719 INFO:tasks.rados.rados.0.trial153.stdout:1164: finishing write tid 5 to trial15342161-34 2026-02-20T12:04:15.719 INFO:tasks.rados.rados.0.trial153.stdout:1164: oid 34 updating version 0 to 203 2026-02-20T12:04:15.720 INFO:tasks.rados.rados.0.trial153.stdout:1164: oid 34 version 203 is already newer than 203 2026-02-20T12:04:15.720 INFO:tasks.rados.rados.0.trial153.stdout:1164: oid 34 version 203 is already newer than 203 2026-02-20T12:04:15.720 INFO:tasks.rados.rados.0.trial153.stdout:1164: oid 34 version 203 is already newer than 203 2026-02-20T12:04:15.720 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 34 v 203 (ObjNum 559 snap 145 seq_num 559) dirty exists 2026-02-20T12:04:15.720 INFO:tasks.rados.rados.0.trial153.stdout:1164: left oid 34 (ObjNum 559 snap 145 seq_num 559) 2026-02-20T12:04:16.528 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~303s 2026-02-20T12:04:16.529 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2026-02-20T12:04:16.539 INFO:tasks.rados.rados.0.trial153.stdout:1164: done (2 left) 2026-02-20T12:04:16.539 INFO:tasks.rados.rados.0.trial153.stdout:1165: done (1 left) 2026-02-20T12:04:16.540 INFO:tasks.rados.rados.0.trial153.stdout:1166: done (0 left) 2026-02-20T12:04:16.540 INFO:tasks.rados.rados.0.trial153.stdout:1167: write oid 14 current snap is 146 2026-02-20T12:04:16.540 INFO:tasks.rados.rados.0.trial153.stdout:1167: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2026-02-20T12:04:16.544 INFO:tasks.rados.rados.0.trial153.stdout:1167: writing trial15342161-14 from 628727 to 1337185 tid 1 2026-02-20T12:04:16.550 INFO:tasks.rados.rados.0.trial153.stdout:1167: writing trial15342161-14 from 2045457 to 2540316 tid 2 2026-02-20T12:04:16.553 INFO:tasks.rados.rados.0.trial153.stdout:1167: writing trial15342161-14 from 3123240 to 3428727 tid 3 2026-02-20T12:04:16.553 INFO:tasks.rados.rados.0.trial153.stdout:1168: snap_create 2026-02-20T12:04:16.594 INFO:tasks.rados.rados.0.trial153.stdout:1167: finishing write tid 1 to trial15342161-14 2026-02-20T12:04:16.602 INFO:tasks.rados.rados.0.trial153.stdout:1167: finishing write tid 2 to trial15342161-14 2026-02-20T12:04:16.609 INFO:tasks.rados.rados.0.trial153.stdout:1167: finishing write tid 3 to trial15342161-14 2026-02-20T12:04:16.615 INFO:tasks.rados.rados.0.trial153.stdout:1167: finishing write tid 4 to trial15342161-14 2026-02-20T12:04:16.617 INFO:tasks.rados.rados.0.trial153.stdout:1167: finishing write tid 5 to trial15342161-14 2026-02-20T12:04:16.617 INFO:tasks.rados.rados.0.trial153.stdout:1167: oid 14 updating version 0 to 377 2026-02-20T12:04:16.618 INFO:tasks.rados.rados.0.trial153.stdout:1167: oid 14 version 377 is already newer than 377 2026-02-20T12:04:16.618 INFO:tasks.rados.rados.0.trial153.stdout:1167: oid 14 version 377 is already newer than 377 2026-02-20T12:04:16.618 INFO:tasks.rados.rados.0.trial153.stdout:1167: oid 14 version 377 is already newer than 377 2026-02-20T12:04:16.618 INFO:tasks.rados.rados.0.trial153.stdout:update_object_version oid 14 v 377 (ObjNum 561 snap 146 seq_num 561) dirty exists 2026-02-20T12:04:16.618 INFO:tasks.rados.rados.0.trial153.stdout:1167: left oid 14 (ObjNum 561 snap 146 seq_num 561) 2026-02-20T12:04:16.703 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-20T12:04:16.780 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:16.775+0000 7f7ca097e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:16.782 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:04:16.778+0000 7f7ca097e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:04:16.811 INFO:teuthology.orchestra.run.trial153.stderr:Error ENXIO: problem getting command descriptions from osd.2 2026-02-20T12:04:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-20T12:04:16.827 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-20T12:04:16.886 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:16.959 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:16.962 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:17.032 INFO:tasks.ceph.osd.0:Sent signal 15 2026-02-20T12:04:17.033 INFO:tasks.ceph.osd.1:Sent signal 15 2026-02-20T12:04:17.033 INFO:tasks.ceph.osd.3:Sent signal 15 2026-02-20T12:04:17.033 INFO:tasks.ceph.mon.a:Sent signal 15 2026-02-20T12:04:17.033 INFO:tasks.ceph.mgr.x:Sent signal 15 2026-02-20T12:04:17.034 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing thrasher OSDThrasher 2026-02-20T12:04:17.034 INFO:tasks.ceph.mon.a.trial153.stderr:2026-02-20T12:04:17.029+0000 7f59a13fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 39930) UID: 0 2026-02-20T12:04:17.034 INFO:tasks.ceph.mon.a.trial153.stderr:2026-02-20T12:04:17.029+0000 7f59a13fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2026-02-20T12:04:17.035 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-20T12:04:17.031+0000 7f4964df8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 39234) UID: 0 2026-02-20T12:04:17.035 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-20T12:04:17.031+0000 7f4964df8640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2026-02-20T12:04:21.963 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:22.036 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:22.039 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:27.041 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:27.113 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:27.115 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:27.116 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:32.117 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:32.188 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:32.192 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:37.192 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:37.264 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:37.267 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:42.269 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:42.341 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:42.344 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:47.345 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:47.418 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:47.422 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:52.423 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:52.497 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:52.501 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:04:57.502 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:04:57.574 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:04:57.576 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:04:57.577 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:05:02.578 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:05:02.649 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:05:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:05:02.652 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:05:07.653 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:05:07.725 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:05:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:05:07.728 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-20T12:05:12.729 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-20T12:05:12.800 INFO:teuthology.orchestra.run.trial183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-20T12:05:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-20T12:05:12.807 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.2 restart 2026-02-20T12:05:12.807 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1384, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.2 restart 2026-02-20T12:05:12.808 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-20T12:05:12.808 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing remote process ceph-test-rados-ceph 2026-02-20T12:05:12.809 DEBUG:tasks.rados:Stopping ceph-test-rados-ceph due to exception The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T12:05:12.809 INFO:tasks.rados:Stopping instance 0 2026-02-20T12:05:12.821 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T12:05:12.823 ERROR:teuthology.run_tasks:Manager failed: rados Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T12:05:12.825 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2026-02-20T12:05:12.840 INFO:tasks.thrashosds:joining thrashosds 2026-02-20T12:05:12.840 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-20T12:05:12.840 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-20T12:05:12.840 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-20T12:05:12.951 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:05:12.944+0000 7fedc2ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:07:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-20T12:07:12.906 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2970, in wait_for_all_osds_up while not self.are_all_osds_up(): ^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2960, in are_all_osds_up x = self.get_osd_dump() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2670, in get_osd_dump return self.get_osd_dump_json()['osds'] ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2662, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2026-02-20T12:07:12.908 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-20T12:07:12.919 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-20T12:07:12.919 DEBUG:teuthology.orchestra.run.trial153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-20T12:07:12.997 INFO:teuthology.orchestra.run.trial153.stderr:2026-02-20T12:07:12.992+0000 7f8060c35640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-20T12:09:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-20T12:09:12.951 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-20T12:09:12.953 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-20T12:09:12.953 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-20T12:09:12.954 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-20T12:09:12.954 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T12:09:12.954 INFO:tasks.ceph.osd.0:Stopped 2026-02-20T12:09:12.954 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-20T12:09:12.954 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T12:09:12.954 INFO:tasks.ceph.osd.1:Stopped 2026-02-20T12:09:12.954 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-20T12:09:12.954 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T12:09:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T12:09:12.954 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 1525, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial183 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2026-02-20T12:09:12.956 INFO:tasks.ceph.osd.2:Stopped 2026-02-20T12:09:12.956 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-20T12:09:12.956 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T12:09:12.957 INFO:tasks.ceph.osd.3:Stopped 2026-02-20T12:09:12.957 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-20T12:09:12.957 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-20T12:09:12.957 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T12:09:12.957 INFO:tasks.ceph.mgr.x:Stopped 2026-02-20T12:09:12.957 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-20T12:09:12.957 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-20T12:09:12.957 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T12:09:12.957 INFO:tasks.ceph.mon.a:Stopped 2026-02-20T12:09:12.957 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-20T12:09:12.957 DEBUG:teuthology.orchestra.run.trial153:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v 'but it is still running' | grep -E -v 'objects unfound and apparently lost' | grep -E -v 'overall HEALTH_' | grep -E -v '\(OSDMAP_FLAGS\)' | grep -E -v '\(OSD_' | grep -E -v '\(PG_' | grep -E -v '\(POOL_' | grep -E -v '\(CACHE_POOL_' | grep -E -v '\(SMALLER_PGP_NUM\)' | grep -E -v '\(OBJECT_' | grep -E -v SLOW_OPS | grep -E -v '\(REQUEST_SLOW\)' | grep -E -v '\(TOO_FEW_PGS\)' | grep -E -v 'slow request' | grep -E -v 'timeout on replica' | grep -E -v 'late reservation from' | grep -E -v MON_DOWN | grep -E -v OSDMAP_FLAGS | grep -E -v OSD_DOWN | grep -E -v PG_DEGRADED | grep -E -v PG_AVAILABILITY | grep -E -v POOL_APP_NOT_ENABLED | grep -E -v 'mons down' | grep -E -v 'mon down' | grep -E -v 'out of quorum' | grep -E -v noscrub | grep -E -v nodeep-scrub | grep -E -v 'Degraded data redundancy' | grep -E -v 'is down' | grep -E -v 'osds down' | grep -E -v 'pg .*? is .*?degraded.*?, acting' | grep -E -v 'pg .*? is stuck' | grep -E -v 'pg degraded' | grep -E -v PG_BACKFILL_FULL | grep -E -v 'Low space hindering backfill .*? backfill_toofull' | grep -E -v OSD_ROOT_DOWN | grep -E -v 'pgs degraded' | grep -E -v 'pgs undersized' | grep -E -v 'is active.*backfill_toofull.*' | head -n 1 2026-02-20T12:09:12.983 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial153.front.sepia.ceph.com 2026-02-20T12:09:12.983 DEBUG:teuthology.orchestra.run.trial153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-20T12:09:13.050 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial153.front.sepia.ceph.com 2026-02-20T12:09:13.050 DEBUG:teuthology.orchestra.run.trial153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-20T12:09:13.116 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial183.front.sepia.ceph.com 2026-02-20T12:09:13.116 DEBUG:teuthology.orchestra.run.trial183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-20T12:09:13.145 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial183.front.sepia.ceph.com 2026-02-20T12:09:13.145 DEBUG:teuthology.orchestra.run.trial183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-20T12:09:13.212 INFO:tasks.ceph:Archiving mon data... 2026-02-20T12:09:13.214 DEBUG:teuthology.misc:Transferring archived files from trial153:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590/data/mon.a.tgz 2026-02-20T12:09:13.215 DEBUG:teuthology.orchestra.run.trial153:> mktemp 2026-02-20T12:09:13.232 INFO:teuthology.orchestra.run.trial153.stdout:/tmp/tmp.y4mU93Z3ll 2026-02-20T12:09:13.232 DEBUG:teuthology.orchestra.run.trial153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.y4mU93Z3ll 2026-02-20T12:09:13.346 DEBUG:teuthology.orchestra.run.trial153:> sudo chmod 0666 /tmp/tmp.y4mU93Z3ll 2026-02-20T12:09:13.428 DEBUG:teuthology.orchestra.remote:trial153:/tmp/tmp.y4mU93Z3ll is 397KB 2026-02-20T12:09:13.524 DEBUG:teuthology.orchestra.run.trial153:> rm -fr /tmp/tmp.y4mU93Z3ll 2026-02-20T12:09:13.541 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-20T12:09:13.541 DEBUG:teuthology.orchestra.run.trial153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-20T12:09:13.583 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-20T12:09:13.660 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-20T12:09:13.676 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-20T12:09:13.677 INFO:tasks.ceph:Archiving crash dumps... 2026-02-20T12:09:13.679 DEBUG:teuthology.misc:Transferring archived files from trial153:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590/remote/trial153/crash 2026-02-20T12:09:13.680 DEBUG:teuthology.orchestra.run.trial153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-20T12:09:13.726 DEBUG:teuthology.misc:Transferring archived files from trial183:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590/remote/trial183/crash 2026-02-20T12:09:13.728 DEBUG:teuthology.orchestra.run.trial183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-20T12:09:13.755 INFO:tasks.ceph:Compressing logs... 2026-02-20T12:09:13.755 DEBUG:teuthology.orchestra.run.trial153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T12:09:13.772 DEBUG:teuthology.orchestra.run.trial183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T12:09:13.830 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-20T12:09:13.830 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-20T12:09:13.830 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39381.log 2026-02-20T12:09:13.831 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39334.log 2026-02-20T12:09:13.831 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-client.admin.39381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2026-02-20T12:09:13.831 INFO:teuthology.orchestra.run.trial183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39381.log.gz 2026-02-20T12:09:13.831 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39486.log 2026-02-20T12:09:13.831 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-client.admin.39334.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39334.log.gz 2026-02-20T12:09:13.831 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-client.admin.39432.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-20T12:09:13.831 INFO:teuthology.orchestra.run.trial183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gz 2026-02-20T12:09:13.831 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-client.admin.39486.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39486.log.gz 2026-02-20T12:09:13.845 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46441.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40371.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49467.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46441.log.gz 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40371.log: -- /var/log/ceph/ceph-client.admin.48765.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40371.log.gz 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:gzip/var/log/ceph/ceph-client.admin.49467.log: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49467.log.gz 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43072.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48765.log.gz 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47036.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46928.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43072.log.gz 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46279.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47036.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47036.log.gz 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45575.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46928.log.gz 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46279.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46279.log.gz 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44099.log 2026-02-20T12:09:13.846 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47306.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45575.log.gz 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44099.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44099.log.gz 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44603.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47306.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47306.log.gz 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45683.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49630.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44603.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44603.log.gz 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43558.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45683.log.gz 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39933.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.49630.log.gz 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41472.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43558.log.gz 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5/var/log/ceph/ceph-client.admin.39933.log: --verbose -- /var/log/ceph/ceph-client.admin.43990.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39933.log.gz 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41472.log: gzip 49.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.41472.log.gz --verbose 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr: -- /var/log/ceph/ceph-client.admin.42477.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43990.log: gzip -5 50.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43990.log.gz /var/log/ceph/ceph-client.admin.45251.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr: 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44207.log 2026-02-20T12:09:13.847 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42477.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40497.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45251.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40274.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44207.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48603.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40497.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40497.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40274.log: gzip -5 50.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40274.log.gz /var/log/ceph/ceph-client.admin.40300.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr: 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48278.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48603.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45197.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44873.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48278.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48278.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46333.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45197.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48062.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44873.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45954.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46333.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41616.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48062.log.gz 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48711.log 2026-02-20T12:09:13.848 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45954.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42423.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41616.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46495.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48711.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42423.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42423.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42207.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48008.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46495.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42207.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42207.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41059.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46008.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48008.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41059.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41059.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43234.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49305.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46008.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43234.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43234.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46549.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43288.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49305.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46549.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46549.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47360.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46062.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.43288.log.gz 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45792.log 2026-02-20T12:09:13.849 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47360.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46062.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46062.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41567.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44317.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45792.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41567.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41567.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41055.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47684.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44317.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42261.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41055.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48386.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47684.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42261.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42261.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45737.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48386.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.48386.log.gz --verbose 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr: -- /var/log/ceph/ceph-client.admin.46711.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42855.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45737.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46874.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46711.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49251.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42855.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47576.log 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46874.log.gz 2026-02-20T12:09:13.850 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43720.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.49251.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47576.log: gzip 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47576.log.gz -5 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45629.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43720.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43720.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45467.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45629.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45629.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44262.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45467.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45467.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40861.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44262.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44262.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46657.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40861.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40861.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44533.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46657.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46657.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45143.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44533.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40636.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47090.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44765.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45143.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47090.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47090.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40427.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47738.log 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44765.log.gz 2026-02-20T12:09:13.851 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40427.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40427.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42693.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47144.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47738.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42693.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42693.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42747.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46603.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.47144.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42747.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42747.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45035.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40685.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46603.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44371.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45035.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40685.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40685.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40864.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48873.log 50.4% 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44371.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45521.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40864.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43504.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48873.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45521.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45521.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40522.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43342.log 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43504.log.gz 2026-02-20T12:09:13.852 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40522.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.40522.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46116.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43342.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43342.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41812.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42036.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.46116.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43018.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41812.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41371.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42036.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43018.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43018.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42963.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48170.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41371.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42963.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47954.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40734.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48170.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47954.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47954.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48495.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44819.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40734.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48495.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48495.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42369.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47252.log 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44819.log.gz 2026-02-20T12:09:13.853 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40587.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42369.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:gzip/var/log/ceph/ceph-client.admin.47252.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43180.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47252.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40587.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40587.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48927.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43180.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43180.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47900.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49089.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48927.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41175.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47900.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49089.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49089.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41861.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41175.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41861.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41861.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42037.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43450.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42037.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42037.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42909.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43774.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43450.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44981.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42909.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43774.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43774.log.gz 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49359.log 2026-02-20T12:09:13.854 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44981.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49359.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49359.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48981.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45305.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48981.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48981.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42161.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42531.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45305.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45305.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.0.42161.log: 50.0% -- replaced with /var/log/ceph/ceph-client.0.42161.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40862.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48819.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42531.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40862.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40862.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41257.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47630.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48819.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41257.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41257.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41058.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46387.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47630.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49197.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41058.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46820.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46387.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41763.log 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49197.log.gz 2026-02-20T12:09:13.855 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46820.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46820.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41060.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41763.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41665.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49143.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41060.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 87.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46766.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41665.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43126.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.49143.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47468.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46766.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43126.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43126.log.gzgzip 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47198.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48332.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47468.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47198.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47198.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43396.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43882.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48332.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43396.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43396.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42639.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45359.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43882.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49593.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42639.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47414.log 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45359.log.gz 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49593.log: 8.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.49593.log.gz -5 2026-02-20T12:09:13.856 INFO:teuthology.orchestra.run.trial153.stderr: --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46982.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47414.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44044.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43936.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46982.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44044.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.44044.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48440.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42315.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43936.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43936.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49035.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48440.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42315.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42315.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44479.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49413.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49035.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44479.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44479.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43828.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42585.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49413.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43828.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43828.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41398.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45089.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.42585.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41398.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41398.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41471.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40019.log 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45089.log.gz 2026-02-20T12:09:13.857 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41471.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41471.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45413.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40019.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38876.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40019.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45413.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45413.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44153.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph.tmp-client.admin.38876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44425.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38876.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44153.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44153.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48116.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44425.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44425.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40863.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44927.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48116.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.40863.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40863.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42801.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48657.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44927.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42801.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42801.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47846.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48549.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48657.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47846.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47846.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43666.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47522.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48549.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43666.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43666.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45900.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46225.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47522.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45900.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45900.log.gz 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41714.log 2026-02-20T12:09:13.858 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48224.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.46225.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41714.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41714.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46171.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.48224.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48224.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.46171.log: 50.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.46171.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49534.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42114.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.49534.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49534.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41252.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.42114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45846.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42114.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41252.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41252.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41255.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr: 87.5% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44657.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.45846.log: 48.4% -- replaced with /var/log/ceph/ceph-client.admin.45846.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41255.log: gzip 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41255.log.gz -5 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41937.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.44657.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44657.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43612.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.41937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47792.log 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41937.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.43612.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43612.log.gz 2026-02-20T12:09:13.859 INFO:teuthology.orchestra.run.trial153.stderr:/var/log/ceph/ceph-client.admin.47792.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47792.log.gz 2026-02-20T12:09:13.860 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-mgr.x.log: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-20T12:09:13.947 INFO:teuthology.orchestra.run.trial153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-20T12:09:14.262 INFO:teuthology.orchestra.run.trial183.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-20T12:09:15.172 INFO:teuthology.orchestra.run.trial153.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-20T12:09:15.357 INFO:teuthology.orchestra.run.trial153.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-20T12:09:15.359 INFO:teuthology.orchestra.run.trial153.stderr: 2026-02-20T12:09:15.359 INFO:teuthology.orchestra.run.trial153.stderr:real 0m1.574s 2026-02-20T12:09:15.359 INFO:teuthology.orchestra.run.trial153.stderr:user 0m2.974s 2026-02-20T12:09:15.359 INFO:teuthology.orchestra.run.trial153.stderr:sys 0m0.210s 2026-02-20T12:09:15.708 INFO:teuthology.orchestra.run.trial183.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-20T12:09:15.710 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-20T12:09:15.711 INFO:teuthology.orchestra.run.trial183.stderr:real 0m1.895s 2026-02-20T12:09:15.711 INFO:teuthology.orchestra.run.trial183.stderr:user 0m2.238s 2026-02-20T12:09:15.711 INFO:teuthology.orchestra.run.trial183.stderr:sys 0m0.132s 2026-02-20T12:09:15.711 INFO:tasks.ceph:Archiving logs... 2026-02-20T12:09:15.712 DEBUG:teuthology.misc:Transferring archived files from trial153:/var/log/ceph to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590/remote/trial153/log 2026-02-20T12:09:15.713 DEBUG:teuthology.orchestra.run.trial153:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-20T12:09:16.184 DEBUG:teuthology.misc:Transferring archived files from trial183:/var/log/ceph to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590/remote/trial183/log 2026-02-20T12:09:16.184 DEBUG:teuthology.orchestra.run.trial183:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-20T12:09:16.464 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial153 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-20T12:09:16.466 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-20T12:09:16.476 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T12:09:16.590 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-20T12:09:16.601 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/contextutil.py", line 34, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e6d9099ce03dc21193fda653159ec82465c3a94a/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_445d5c9ba18a4040e70e25303230c36c77773acb/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T12:09:16.602 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-20T12:09:16.602 DEBUG:teuthology.orchestra.run.trial153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-20T12:09:16.604 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-20T12:09:16.629 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-20T12:09:16.639 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-20T12:09:16.639 DEBUG:teuthology.orchestra.run.trial153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T12:09:16.670 DEBUG:teuthology.orchestra.run.trial183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T12:09:16.682 INFO:teuthology.orchestra.run.trial153.stderr:bash: line 1: ntpq: command not found 2026-02-20T12:09:16.685 INFO:teuthology.orchestra.run.trial183.stderr:bash: line 1: ntpq: command not found 2026-02-20T12:09:17.048 INFO:teuthology.orchestra.run.trial153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-20T12:09:17.048 INFO:teuthology.orchestra.run.trial153.stdout:=============================================================================== 2026-02-20T12:09:17.048 INFO:teuthology.orchestra.run.trial153.stdout:^* 144.202.66.214.vultruser> 3 6 377 32 -1112us[-1313us] +/- 31ms 2026-02-20T12:09:17.048 INFO:teuthology.orchestra.run.trial153.stdout:^+ ntp.silvertree.org 1 6 377 31 +2424us[+2424us] +/- 40ms 2026-02-20T12:09:17.049 INFO:teuthology.orchestra.run.trial153.stdout:^+ ntp.maxhost.io 2 6 377 162 -356us[ -560us] +/- 55ms 2026-02-20T12:09:17.049 INFO:teuthology.orchestra.run.trial153.stdout:^- tick.chi1.ntfo.org 3 6 177 20 -1452us[-1452us] +/- 149ms 2026-02-20T12:09:17.050 INFO:teuthology.orchestra.run.trial183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-20T12:09:17.051 INFO:teuthology.orchestra.run.trial183.stdout:=============================================================================== 2026-02-20T12:09:17.051 INFO:teuthology.orchestra.run.trial183.stdout:^* 144.202.66.214.vultruser> 3 6 377 32 -720us[ -681us] +/- 31ms 2026-02-20T12:09:17.051 INFO:teuthology.orchestra.run.trial183.stdout:^+ ntp.silvertree.org 1 6 377 34 +5677us[+5716us] +/- 44ms 2026-02-20T12:09:17.051 INFO:teuthology.orchestra.run.trial183.stdout:^+ ntp.maxhost.io 2 6 357 32 +157us[ +196us] +/- 56ms 2026-02-20T12:09:17.051 INFO:teuthology.orchestra.run.trial183.stdout:^- tick.chi1.ntfo.org 3 6 377 98 -1580us[-1544us] +/- 148ms 2026-02-20T12:09:17.051 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-20T12:09:17.069 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-20T12:09:17.069 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-20T12:09:17.085 DEBUG:teuthology.orchestra.run.trial153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-20T12:09:17.146 DEBUG:teuthology.orchestra.run.trial183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-20T12:09:17.201 DEBUG:teuthology.orchestra.run.trial153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-20T12:09:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T12:09:17.230 DEBUG:teuthology.orchestra.run.trial183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-20T12:09:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T12:09:17.257 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-20T12:09:17.274 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-20T12:09:17.290 INFO:teuthology.task.internal:Duration was 1114.692227 seconds 2026-02-20T12:09:17.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-20T12:09:17.306 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-20T12:09:17.306 DEBUG:teuthology.orchestra.run.trial153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-20T12:09:17.309 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-20T12:09:17.346 INFO:teuthology.orchestra.run.trial153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-20T12:09:17.346 INFO:teuthology.orchestra.run.trial183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-20T12:09:17.597 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-20T12:09:17.597 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial153.front.sepia.ceph.com 2026-02-20T12:09:17.597 DEBUG:teuthology.orchestra.run.trial153:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-20T12:09:17.615 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial183.front.sepia.ceph.com 2026-02-20T12:09:17.615 DEBUG:teuthology.orchestra.run.trial183:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-20T12:09:17.657 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-20T12:09:17.657 DEBUG:teuthology.orchestra.run.trial153:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T12:09:17.659 DEBUG:teuthology.orchestra.run.trial183:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T12:09:17.760 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-20T12:09:17.760 DEBUG:teuthology.orchestra.run.trial153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T12:09:17.763 DEBUG:teuthology.orchestra.run.trial183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T12:09:17.785 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T12:09:17.785 INFO:teuthology.orchestra.run.trial153.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T12:09:17.785 INFO:teuthology.orchestra.run.trial153.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T12:09:17.785 INFO:teuthology.orchestra.run.trial153.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-20T12:09:17.785 INFO:teuthology.orchestra.run.trial153.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-20T12:09:17.786 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T12:09:17.787 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T12:09:17.787 INFO:teuthology.orchestra.run.trial183.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T12:09:17.787 INFO:teuthology.orchestra.run.trial183.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-20T12:09:17.787 INFO:teuthology.orchestra.run.trial183.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-20T12:09:17.795 INFO:teuthology.orchestra.run.trial153.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-20T12:09:17.797 INFO:teuthology.orchestra.run.trial183.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.7% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-20T12:09:17.799 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-20T12:09:17.816 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-20T12:09:17.816 DEBUG:teuthology.orchestra.run.trial153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-20T12:09:17.863 DEBUG:teuthology.orchestra.run.trial183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-20T12:09:17.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-20T12:09:17.901 DEBUG:teuthology.orchestra.run.trial153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-20T12:09:17.904 DEBUG:teuthology.orchestra.run.trial183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-20T12:09:17.925 INFO:teuthology.orchestra.run.trial153.stdout:kernel.core_pattern = core 2026-02-20T12:09:17.952 INFO:teuthology.orchestra.run.trial183.stdout:kernel.core_pattern = core 2026-02-20T12:09:17.965 DEBUG:teuthology.orchestra.run.trial153:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-20T12:09:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T12:09:17.999 DEBUG:teuthology.orchestra.run.trial183:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-20T12:09:18.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T12:09:18.020 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-20T12:09:18.038 INFO:teuthology.task.internal:Transferring archived files... 2026-02-20T12:09:18.039 DEBUG:teuthology.misc:Transferring archived files from trial153:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590/remote/trial153 2026-02-20T12:09:18.039 DEBUG:teuthology.orchestra.run.trial153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-20T12:09:18.076 DEBUG:teuthology.misc:Transferring archived files from trial183:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/jjperez-2026-02-20_10:21:31-crimson-rados-wip-perezjos-crimson-only-19-02-2026-PR64439-distro-debug-trial/60590/remote/trial183 2026-02-20T12:09:18.076 DEBUG:teuthology.orchestra.run.trial183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-20T12:09:18.112 INFO:teuthology.task.internal:Removing archive directory... 2026-02-20T12:09:18.113 DEBUG:teuthology.orchestra.run.trial153:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-20T12:09:18.115 DEBUG:teuthology.orchestra.run.trial183:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-20T12:09:18.161 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-20T12:09:18.179 INFO:teuthology.task.internal:Not uploading archives. 2026-02-20T12:09:18.180 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-20T12:09:18.196 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-20T12:09:18.197 DEBUG:teuthology.orchestra.run.trial153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-20T12:09:18.200 DEBUG:teuthology.orchestra.run.trial183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-20T12:09:18.215 INFO:teuthology.orchestra.run.trial153.stdout: 917521 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 12:09 /home/ubuntu/cephtest 2026-02-20T12:09:18.219 INFO:teuthology.orchestra.run.trial183.stdout: 917572 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 12:09 /home/ubuntu/cephtest 2026-02-20T12:09:18.220 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-20T12:09:18.238 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-20T12:09:18.256 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: BarkError: The WatchDog barked due to Daemon ceph.osd.2 has failed 2026-02-20T12:09:18.261 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/segmented$/{crimson_seastore_segmented_2q} thrashers/{default} thrashosds-health workloads/snaps-few-objects} duration: 1114.6922268867493 failure_reason: The WatchDog barked due to Daemon ceph.osd.2 has failed flavor: debug owner: scheduled_jjperez@soko04.front.sepia.ceph.com status: fail success: false 2026-02-20T12:09:18.261 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-20T12:09:18.469 INFO:teuthology.run:FAIL